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ABSTRACT 


With spacecraft designs placing more emphasis on reduced cost, faster design 
time, and higher performance, it 1s easy to understand why more commercial-off-the- 
shelf (COTS) devices are being used in space based applications. The COTS devices 
offer spacecraft designers shorter design-to-orbit times, lower system costs, orders of 
magnitude better performance, and a much better software availability than their radiation 
hardened (radhard) counterparts. The major drawback to using COTS devices in space is 
their increased susceptibility to the effects of radiation, single event upsets (SEUs) in 
particular. 

This thesis will focus on the implementation of a fault tolerant computer system. 
The hardware design presented here has two different benefits. First, the system can act 
as a software testbed, which allows testing of software fault tolerant techniques in the 
presence of radiation induced SEUs. This allows the testing of the software algorithms in 
the environment they were designed to operate in without the expense of being placed in 
orbit. Additionally, the design can be used as a hybrid fault tolerant computer system. 
By combining the masking ability of the hardware with supporting software, the system 
can mask out and reset processor errors in real ttme. The design layout will be presented 


using OrCAD® schematics. 
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I. INTRODUCTION 


The decline of Department of Defense (DOD) funding of research into radiation 
hardening and reliability of microelectronics since the end of the Cold War has had 
serious impact on the price and availability of these parts. This, coupled with reductions 
in the space budget, has spacecraft designers looking for alternative ways to implement 
reliable systems with reduced cost, faster design time, and higher performance. One 
alternative is the use of commercial-off-the-shelf (COTS) devices in place of radiation 
hardened devices. COTS devices offer spacecraft designers shorter design-to-orbit times, 
lower system costs, orders of magnitude better performance, and a much better software 
availability than their radiation hardened (radhard) counterparts. The major drawback to 
using COTS devices in space is their increased susceptibility to the effects of radiation, 
both total dose and single event upsets (SEUs). 

This thesis will present the implementation of a fault tolerant computer system. 
The hardware design presented here has two different benefits. First, the system can act 
as a software testbed, which enables the testing of software fault tolerant techniques in the 
presence of radiation induced SEUs. This allows the testing of the software algorithms in 
the environment they were designed to operate in without the expense of being placed in 
orbit. Additionally, the design can be used as a hybrid fault tolerant computer system. 
By combining the masking ability of the hardware with supporting software, the system 
can mask out and reset single processor errors with small cost in time. Both of these 


concepts will be discussed further in the body of the thesis. 


A. THE SPACE ENVIRONMENT 


The most important aspect to the designer of an electronic system is its function. 
Another vital interest to the system designer is the environment the system is going to 
operate in. If the environment is going to be space, then there are many factors that must 
be taken into account. The uninformed might think space 1s a benign environment, but 
that can not be farther from the truth. The vacuum, temperature extremes, debris, and 
radiation all interact with the system in one way or another and special efforts must be 
taken to account for their negative side effects. 


I Vacuum 

As you move away from the Earth’s surface, the number of particles per unit 
volume continues to decrease. Starting at roughly 10'° particles per cubic centimeter 
(cm”) on the surface of the Earth, this number decreases drastically as you increase in 
altitude. The average particle density from 600 to 1200 miles 1s only about 100 particles 
per cm>. Additionally, as with the particle density, the pressure associated with an 
altitude decreases as you move away from the surface of the Earth. While the pressure is 
near 760 millimeters of mercury (mmHg) at the surface of the earth, it is only about or 
to 10'° mmHg past 1200 miles. These low pressures are sometimes referred to as hard 
vacuum. [Ref. |] 

One phenomena caused by the hard vacuum of space is called outgassing. 
Outgassing occurs when a material actually loses mass because molecular gasses trapped 


inside or on the surface of the material at ground level are pulled out and away by the 


vacuum of space. A byproduct of outgassing is the cold weld. In the atmosphere, metal 


to 


surfaces have a thin layer of the molecular gas, which acts as a lubricant when two 
surfaces are in contact. When the vacuum removes the gas, the metal surfaces are 
allowed to touch and they will bond together. This issue is one to be dealt with by a 
materials scientist. The solution to this problem usually relies on the selection of 
appropriate materials to minimize the effects of the vacuum. [Ref. 1] 


2. Meteoroids 


Meteoroids and orbital debris pose a risk to satellites. Although there are a large 
number of meteoroids and space dust near the Earth, most are fairly small and do not pose 
a large risk of catastrophic damage to a satellite. There have been a couple of satellites 
that have mysteriously quit working and the conjecture is that the failure might be due to 
damage caused by impact with meteoroids. With this in mind, the system should be 
designed to deal with the higher probability of micrometeoroid strikes, where only a small 
portion of the system is damaged. One option is to design the system using redundancy, a 
concept ies will be explored later in this chapter. (Ref. 1] 


3 Temperature 


The Earth’s atmosphere and the mass of the planet keep the ambient temperatures 
within a fairly moderate range at the surface of the planet. That can not be said of the 
space environment. Careful considerations with respect to thermal radiation must be 
taken into account when designing systems for space application. The skin of a 
spacecraft exposed to the sun will rise to a very high temperature while the shaded side 
falls to an extremely low temperature. This causes a very high temperature difference 


across the space vehicle. With that in mind, microelectronic devices used in systems 


designed for space must function correctly over a much larger temperature range than 
required within the Earth’s atmosphere. [Ref. 1] 


4, Radiation 


Radiation is the emission or propagation of waves or particles. It is the key 
element of the space environment that our design is taking into consideration. High- 
energy charged particles can cause damage or disruptions, which are discussed later in the 
chapter, in microelectronic devices. These particles are either ions or photons. Ions, 
except for the neutron, have both a charge and mass associated with them. There are 
basically four types of ions: electrons, neutrons, light, and heavy. Light ions have a very 
low mass, such as protons, which are hydrogen atoms with the electron missing, and 
alpha particles, which are helium atoms with both the electrons stripped off. Heavy ions 
are any element heavier than helium with one or more electrons missing. Unlike ions, 
photons have neither mass nor charge. They are just very short wavelengths of light, such 
as X-rays and gamma rays. [Ref. 2] 

There are several contributors to the radiation effects near the Earth. The largest 
contributor to a device’s total dose is from particles trapped in the Earth’s geomagnetic 
field. These trapped particles make up an area known as the Van Allen Belts. Any 
satellite in orbit 1s subject to effects from the Van Allen Belts. Another contnbutor is 
solar particles. Due to the high temperatures of the sun, many particles have enough 
energy to escape the sun’s gravity. Those particles continuously flow across the Earth in 
What 1s called the Solar Wind. The sun continually moves through cycles of solar 


activity, where a single cycle takes approximately eleven years to complete. Another 


source of radiation is galactic cosmic rays. These are heavy ions produced by events, 
such as exploding stars, outside our solar system. [Ref. 2] 

When radiation interacts with microelectronic devices, it is either absorbed into or 
passes through the semiconductor leaving a path of ionization. The radiation has four 
major different types of effects on the semiconductor: Total Dose Effects, Dose Rate 
Effects, Displacement Damage, and Single Event Effects (SEEs). [Ref. 2] 

Total Dose Effects are device failures caused by the lifetime sum of radiation 
absorbed by the device. Similarly, Dose Rate Effects are where the device fails to 
function at a particular radiation rate, or number of particles per unit of time. 
Displacement Damage deals with nuclear interactions between the particle and the 
semiconductor. The radiation changes the nuclear makeup of crystal atoms within the 
semiconductor, which leads to device failure. [Ref. 2] Finally, SEEs are the effects of 
charged particles changing the state of transistors in the circuit. SEEs are the radiation 
effect we are most interested in and are discussed in greater detail in the following 


section. 


B. SINGLE EVENT EFFECTS (SEE) 


The main difference between SEEs and the radiation effects discussed in the last 
section 1s that SEEs are not necessarily destructive in nature. SEEs are caused when an 
ionized particle deposits enough charge to cause transistors to change state in a device. In 
most cases, the transistor only changes state long enough for the charge to be absorbed 
back into the system and then resumes its original state. The transistor’s state change can 


lead to latchup in parasitic transistors, be purely transient, or be latched into a storage 


element. These three main types of SEE in Complimentary Metal Oxide Semiconductors 
(CMOS), which is currently the most popular type of circuit implementation for 
microelectronic devices, are discussed in the following sections. [Ref. 2] 


de Single Event Latchup (SEL) 


When CMOS field effect transistors are fabricated near each other on a single 
chip, one of the unwanted byproducts 1s a pair of vertical bipolar junction transistors. An 
SEL is caused when a charged particle passes close enough to this circuit to bias the two 
parasitic transistors on. This creates a very low impedance path to ground, which has two 
possible outcomes. If the current drawn through the parasitic transistors generates more 
heat than the device can dissipate, it will be destroyed. Even if the device can dissipate 
the heat, the large amount of current drawn through the parasitic transistors prevents the 
remainder of the circuit from operating correctly, which is a non-destructive SEL. The 
normal manifestation of a non-destructive SEL is that of a hung system, which requires a 
system reset for recovery. [Ref. 3] 


a. Single Event Transient (SET) 


Single Event Transients are unexpected, short duration changes in the output 
value of a eerie logic circuit due to the influence of a charged particle. SETs are 
not detrimental to the system. They only adversely effect the system if the pulse change 
is near the clocking edge and is long enough to meet the setup and hold times of the next 
storage unit in the cascade of stages. If the SET meets these criteria, then 1t manifests 


itself as a single event upset (SEU). [Ref. 4] 
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3: Single Event Upsets (SEU) 


An SEU is any unwanted value change in a memory cell, whether it be a latch, 
register, or cache cell, that is caused by charge introduced into the circuit by radiation. In 
microprocessors, SEUs are usually categorized into one of two error types: program flow 
errors and data errors. Program flow errors are errors that occur in the program counter 
(PC), control logic, or any other register that deals with the state of the processor. Data 
errors are usually confined to the registers and data cache. These two types of errors are 
not necessarily exclusive. A data error could occur in a register that is later used as a 
jump address. When the PC jumps to the address held in that register it is in the wrong 
location and begins to execute incorrect code. [Ref. 4] 


c. COMMERCIAL-OFF-THE-SHELF VS. RADIATION HARDENED 
DEVICES 


The radiation effects discussed in the previous sections, with the exception of 
SETs and SEUs, are destructive in nature. The main way of reducing their effects is by 
using radiation hardened (radhard) devices or providing shielding. A radhard device is 
one that is specifically designed to be able to withstand higher amounts of radiation than 
standard commercial parts. 

In a direct comparison of commercial-off-the-shelf (COTS) parts to radhard parts, 
the first question someone is bound to ask is ‘why would we use non-hardened parts in 
such a harsh environment?’ This section gives some insight into the answer of that 


question. 


Ih, Cutting Edge Technology 


As alluded to in the introduction, companies developing and marketing radhard 
devices are on the decline. Without DOD budget support for research into this area, there 
is not sufficient demand from commercial sector customers to motivate companies to sink 
large sums of money into research on these devices. For those reasons, the technology of 
radhard devices is lagging behind state of the art technology by up to two or more 
generations. As an example, a state of the art processor, the 600 MHz Intel Pentium I, 
AMD K-6 IJ, or a RISC design, depending on your preferences, is available off the 
commercial shelf. The nearest competitor on the radiation hardened shelf would be a 66 
MHz 486 processor. This is a whole order of magnitude difference in processor 
capability. [Ref. 5] 

2 Faster Design-to-Orbit Time 

Because many manufacturers of radhard devices offer them as a secondary 
market, many do not readily stock the parts. This causes a delay in the design and test 
phase of the project. With a move to COTS devices, the order delay is completely erased. 
Most of these devices are available from multiple vendors, which gives the designer even 
more leverage in selecting a vendor. Additionally, data on radiation testing for many 
devices is now becoming available to help the designer make an informed decision on the 
correct part to use based on the environment it is to be placed in. [Ref. 5] 


3: Reduced Cost 


Profit or lack thereof is one of the main reasons companies have moved away 


from the manufacture of radiation hardened devices. The low demand for these devices 


OS 


keeps the price hundreds of times higher than the commercial models for several different 
reasons. Since radiation hardened devices employ different techniques in their design to 
reduce the susceptibility to effects from ionized particles, they tend to be larger than the 
non-hardened devices. This lowers the number that will fit on the wafer in the first place 
and increases the probability that the devices will have defects. The reduction in yield 
can be attributed to two main factors. The first is the higher probability of defects in each 
device mentioned before and the second is from the limited number of runs of the 
fabrication process. Because yield normally increases with the number of production runs 
and radhard devices account for a small number of fabrication runs, radhard devices 
normally do not show large improvements in yield. Since the demand for commercial 
devices is much higher, the manufacturer can adjust the fabrication process over a large 
number of runs to increase the yield, which results in lower cost to the consumer. [Ref. 2] 


D. PURPOSE 


The goal of this research is the implementation of a fault tolerant computer system 
using COTS microprocessors that 1s able to accurately compute in the presence of 
radiation induced SEUs. This research specifically concentrates on the ability of the 
system to detect and mark or correct SEUs in a microprocessor in real time. This work 
does not address error detection and correction (EDAC) of memory systems, which has 
been previously researched [Ref. 6] and is a topic left for future inclusion. 

There are two major benefits from this study. First, the system can act as a fault 
tolerant software testbed. The processor is monitored for an SEU. When one is detected, 


a time stamp is generated and the Operating System is observed in its handling of the 


error. This allows the testing of the software algorithms in the environment they were 
designed to operate in without the expense of being placed in orbit. 

The system can also be used as a hybrid fault tolerant computer system. In this 
use, the processor is also monitored for an SEU. When one is detected, the faulty data is 
corrected in the processor and it continues to execute its instructions from the corrected 
data. A major advance in this implementation is the ease with which the error can be 
corrected. The normal mode from previous research in these types of systems has been to 
just reset the processor when an SEU was detected, effectively loosing all computations 
done up to the time of the SEU. 


EB. THESIS ORGANIZATION 


The organization of this thesis follows closely to the design approach used in 
developing the system. Chapter I has been a brief introduction into the environment that 
the system will be operating in. Chapter I 1s background material on research that set the 
foundation for this design. Chapter II presents the hardware design of the system and 
points out changes from the previous design. Chapter IV contains the design of the 
programmable elements of the system, which include the Voter Modules and the System 
and Memory Controllers. Chapter V presents the steps that were taken to have the 
printed circuit board fabricated and the design review that was conducted after the board 
was manufactured. Chapter VI presents the conclusions developed during this research 


and discusses topics for follow-on work. 


Il. BACKGROUND 

The study of fault tolerant computing systems has been going on for many years. 
In fact, many early analog computer systems were designed with duplicate processing 
units because of their propensity to have errors. The EDVAC computer, which was 
designed in 1949, had duplicate Arithmetic Logic Units (ALUs) that continued to 
compute as long as their results agreed. As computer systems moved further into the 
digital era and became more reliable, fault tolerant designs took a back seat to designs 
that improved performance and speed. When computers started to perform critical tasks 
in systems, designers had to once again focus on fault tolerance. [Ref. 7] 

The purpose of this chapter is to provide the reader a brief background for this 
project and the key issues that make it important. The chapter starts by describing the 
general concept of redundancy and focuses in on the design from which this system was 
implemented. 


A. REDUNDANCY 


The fundamental concept for implementing fault tolerance is redundancy. In its 
most basic form, redundancy is the addition of resources beyond those required by the 
system for normal operation to achieve reliability, availability, or safety. There are four 
different kinds of redundancy that can be implemented in a system. They are software, 
information, time, and hardware redundancy, each of which is discussed below. [Ref. 7] 


1. Software Redundancy 


One of the key attributes of software redundancy is that it requires a minimal 


amount of additional hardware to support it. Three typical software redundancy 


techniques are consistency checks, capability checks, and n-version programming. 
Consistency checks use a-prior: knowledge of the data set to check it to ensure that it is 
consistent with the expected values. Capability checks will test functional units of the 
system to ensure they are in working order. While both of these methods are software 
tests of hardware, n-version programming Is a software test of software. Different 
programmers code different versions of the same function. All of the versions are 
executed and their results compared to ensure that all the results are the same. [Ref. 7] 


2: Information Redundancy 


Information redundancy provides additional bits along with the data to allow for 
checking that the correct values are received at the destination. The additional 
information can be small enough to allow error detection or robust enough to allow 
multiple bit corrections. Two good examples of information redundancy are the parity bit 
and the Hamming Code. The parity bit is a single bit appended to the data set that allows 
for error detection. Parity can either be even or odd. If even parity is selected, the 
number of ones in the data set and the parity bit should add up to an even number. If odd 
parity 1s selected, the ones should add up to an odd number. If they do not, there is an 
error in the information. The limit of the parity bit 1s its inability to determine which bit 
is in error. The Hamming Code appends multiple bits to the data set, which allows for 
error detection and correction by pointing out the faulty bit. The additional bits appended 
by the Hamming Code are actually weighted parity bits. By determining which Hamming 


Code bits are in error, the faulty data bit can be determined and corrected. [Ref. 7] 


3. Time Redundancy 


One of the costs associated with all forms of redundancy is the large overhead. 
This overhead can be paid in the form of additional hardware, weight, and power 
consumption. In some cases, these assets are not available for trade. When that happens, 
another form of fault tolerance must be found. In systems where speed is not a critical 
issue, time redundancy can be used. The premise behind time redundancy is that the 
same calculations can be performed serially multiple times and the outputs compared to 
check for errors. This requires the system to save the state before the calculation, perform 
the calculation and save it, make a context switch back to the beginning of the 
calculation, perform it again, and then compare the results of the two different 
calculations. [Ref. 7] 


4. Hardware Redundancy 


When you mention redundancy to people, hardware redundancy is typically the 
first thing that enters their mind. This form of redundancy 1s regaining popularity as the 
cost for replicating hardware is getting less and less expensive. There are basically three 
forms of hardware redundancy: passive, dynamic, and hybrid. Passive redundancy uses 
fault masking to keep the fault from propagating out of the current process. This is 
implemented using multiple modules and voting hardware. It does not require any 
actions from the system or operator. Dynamic redundancy monitors the outputs of the 
modules looking for faulty units. When one is detected, the system removes the faulty 
unit from the system and replaces it with a good one if it is available. The hybrid 


approach uses portions of both the passive and dynamic approaches. [Ref. 7] 
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A subset of passive hardware redundancy is the Triple Modular Redundant (TMR) 
system. As the name implies, the TMR system takes the outputs of three replicated 
modules and compares them in a voting unit. The voting unit passes the most common 
input to the output, essentially masking out any single fault. The heart of a TMR system, 


the voting unit, and its truth table are shown below in Figure 2.1 and Table 2.1. 


___———T. Our 


Figure 2.1. 3-Bit Majority Voter Logic Diagram. After Ref. [7] 
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Table 2.1. 3-Bit Majority Voter Truth Table 


The table shows that anytime two or more inputs have the same logic value, either 
a zero or One, then that value is propagated to the output. This system can also be 
extended to an N input model. As N gets larger, the logic required to realize the circuit 


and the added levels of delay get prohibitively large. Typically N is held to three or five. 


3). TMR Implementations 

As stated before, by placing the three modules 1n parallel and voting their outputs, 
a TMR circuit can be implemented. The basic TMR circuit is shown below in Figure 2.2. 
The inputs and output of the modules do not have to be single values. There can be X 
inputs and Y outputs associated with each module. All that needs to be done is place Y 


single bit voters in parallel connected such that they vote each of the respective Y 


outputs. 


Input A 


Input B 


Figure 2.2. Basic TMR Circuit Implementation. After Ref. [7] 





The single voter on the output of the TMR system is a single point of failure. 
That is, if the voter fails and generates or propagates an error, then the error will be 
propagated throughout the circuit. This could become a major problem in cascaded 
circuits. To solve this problem, the voters at the end of each stage can also be triplicated. 


[Ref. 7] An example of this is shown in Figure 2.3. 


9 TO 


Input B Mod 2 SAC 


Input C Va 


Figure 2.3. Triplicated TMR Voters. After Ref. [7] 


B. SOFTWARE VS HARDWARE REDUNDANCY 


The technology associated with very large-scale integrated circuit (VLSI) design 
has advanced at a phenomenal pace. Systems that used to be produced as multiple chip 
designs are now compacted into systems on a chip. Much of the current hardware design 
emphasis is placed on reducing the transistor size in order to increase the capability and 
decrease communication times within the chip. Since the communication times between 
chips in a system has not progressed at the same rate as internal communication, the 
addition of logic layers between the processor and its peripherals would require the 
system bus clock to operate at a reduced frequency. Because of this, general trends have 
moved the reliability issue to software. 

When using software fault tolerant techniques, as in Checkpointing, the software 
must run the same segment of code a minimum of two times. If the results of the first 
two runs match, the processor can continue where it left off after it has saved its internal 
state. If the results do not match, the processor must run the code segment an additional 


time to try to determine which of the first two runs were correct and then save the correct 


internal state. This occupies over half of the processor’s execution cycles in context 
switching and rerunning code. [Ref. 8] 

The option being investigated in this research is a move back to hardware 
redundancy. By applying the TMR technique discussed earlier to the microprocessor, the 
outputs of the processors can be combined through a voter. If an error occurs in one of 
the processors the voter will mask it out and the bus cycle will complete normally. The 
two good processors will then reset the errant processor to their good state and then all 
three processors will continue executing the program. 

A direct comparison of the two options is very difficult. Although the TMR 
design looks like it can save over half the time required to perform a function, it can not 
operate at the same clock speed that the Checkpoint model can. The TMR system has to 
place additional logic between the processor and the memory space to perform the voting 
operation, which decreases the maximum clock speed at which the processor can operate. 


Cc TMR MICROPROCESSOR DESIGN 


The framework for the system implemented in this work was originally designed 
and simulated using Verilog by Lieutenant John C. Payne, Jr., USN, as a Fault Tolerant 
Computing Testbed. The remainder of this section is a brief synopsis of his TMR 
Testbed Design. For a more detailed explanation, please see Ref. [9]. 


ie Processor Selection 


Lt. Payne began his design with the logical step of selecting a microprocessor. In 
his selection process he took into account such factors a COTS vs. Radhard, CISC vs. 


RISC, size, pinout, power, bus width, memory size, speed, and multiple chip vs. single 
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chip implementations. His research led him to select the MIPS R3081 RISController 
produced by Integrated Device Technologies (IDT). This device was selected over 
AMD’s AM29000 and AM29050; IBM and Motorola’s PowerPC 603e, 604e, and 750; 
and IDT’s R36100, R4650, and R5000. A detailed description of the R3081 can be found 
in Ref. [10]. 

2 Hardware Design 

With the processor selected, the next step was to integrate all the peripheral 
components required for the R3081 to function as a TMR computer system. For 
comparison purposes, a block diagram of a single processor system and a TMR system 
are provided on the following page in Figure 2.4 and Figure 2.5, respectively. 

From comparison of Figures 2.4 and 2.5, the architecture of the TMR 
implementation has relatively few changes from the single processor design. The major 
additions are CPU B and CPU C along with the Address, Data, and Control Voters. 

The processors are connected in such a = that the Operating System acts as if 
there is only one processor in the system. The processors are kept in lock step from boot 
up by executing the same instructions in parallel. The processors Address, Data. and 
Control busses are then routed to their respective voters. The voters perform a majority 
vote on the signals and pass them on to the Memory Space and Memory Controller as in a 
single processor system. If an error is detected in a voter, the Memory Controller 


generates an interrupt. [Ref. 9] 
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Figure 2.4. Simple R3081 Board Design. After Ref. [11] 
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Figure 2.5. TMR R3081 Board Design. From Ref. [9] 
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3. Software Design 


The voters allow the fault generated in one of the processors to be masked out and 
the bus cycle to complete correctly, but that does not completely remedy the situation. 
The processor that had the error is now out of synchronization with the other two 
processors. That is where the software design of the Interrupt Handler comes into play. 

When a voter detects a miss compare on its inputs, it signals the Memory 
Controller, which asserts an interrupt input to the processors. The Interrupt Handler 
resets the invalid processor in a very simplistic way. It starts by saving the processor’s 
internal registers to memory. Since all three processors will be executing the instructions, 
the Data Voter will mask out the invalid data from the corrupted CPU. The Interrupt 
Handler then reloads the processors internal registers from memory. This puts the 
corrupted processor back into synchronization with the other two processors. The 
Interrupt Handler then acknowledges the interrupt and returns from the exception. The 
processors then continue execution with the next instruction. 

In order to determine which processor was corrupted, the internal registers of each 
processor must be examined. The data must be captured prior to being voted or the error 
is lost. By placing First-In-First-Out (FIFO) Registers on the address, control, and data 
busses between the processors and voter, each processors internal state can be captured in 


its corresponding FIFO. The arrangement is shown below in Figure 2.6. 
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Figure 2.6. TMR FIFO Interface. From Ref. [9] 
Lt. Payne’s design and simulations prove the fundamental concept of a working 
TMR R3081 system. The next logical step is to implement the system in hardware. The 
next two chapters examine the transition of this design from simulation to hardware. 
Chapter UI presents the hardware implementation, while Chapter IV presents the 
programmable logic synthesis. Because of hardware limitations and additional system 


requirements, the transition 1s not a one-to-one process. 
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II. HARDWARE DESIGN SPACE 


The process of designing a complicated electronic system is very seldom linear in 
nature. That is, the system originally conceived is seldom designed, simulated, 
implemented, and manufactured without changes. Many factors affect the process as the 
design matures, which can cause changes in areas of the design that were once thought to 
be complete. This results in feedback loops in the design process. This design has not 
been an exception to these effects. Jt has gone through several revisions in the 
implementation and manufacturing phases due to feedback from several different factors, 
including changes in requirements and parts availability. This chapter presents the final 
hardware implementation and explains design changes where major deviations from the 
Simulation model were required. 


A. DESIGN PARAMETERS 


As discussed in Chapter I, the most important aspect of the system the designer 
must be familiar with is its function. Chapter If presented a simulated design by Lt. John 
Payne, Jr. [Ref. 9] that fulfilled all the functional requirements, but focused primarily on 
validating the concept of TMR using the IDT R3081 microprocessor. The next step in 
the design process is the hardware implementation. This implementation was developed 
by making modifications to Lt. Payne’s design and using three general functional 
applications: use as an Evaluation Board, easy adaptation to a space flight board, and the 


ability to be tested in a particle accelerator. 


i Evaluation Board 


The evaluation board design parameter adds value by introducing flexibility and 
visibility to the design. For flexibility, socketed parts and programmable logic are used. 
By using socketed parts instead of parts that must be soldered to the printed circuit board 
(PCB), the designer enables the user to quickly and easily exchange integrated circuit (IC) 
chips. This allows easier replacement of damaged parts and requires fewer vias when 
routing traces on the PCB layout, when compared to surface mount devices. Upgrades to 
the system can also be easily incorporated if the new IC chip is pin compatible with the 
old one. The most common use of this advantage would be upgrading to faster parts. 

Using programmable logic to interface the processors and the peripherals is 
another tool the system designer can use to increase flexibility. Unless the same 
manufacturer builds the processor and peripheral, there is normally a requirement for 
additional logic to be placed between the two components to allow them to function 
together properly. The additional logic is usually referred to as “glue logic.” By using a 
programmable device to perform the glue-logic function, the system can easily be 
expanded to incorporate system updates and changes. 

Since evaluation boards are normally used to investigate new concepts, it 1s 
important to be able to capture data from as many internal signal lines as possible. 
Today’s PCBs are normally fabricated using multi-layer designs, making it difficult to 
capture signals residing on internal routes. Connecting the essential signals to a test 


connector, which makes them visible to the user, can alleviate this problem. 


De Space Flight Board 


Although it may seem that designing the initial implementation of a system with 
such a futuristic goal would increase the complexity of the design, in this case it actually 
narrowed the scope on many of the part choices. The premise of this design parameter 
was to focus on parts, other than the microprocessor, that were available in both 
commercial and rad-hard versions or parts that showed a reduced susceptibility to SEUs. 
The pin compatibility and exact functionality between the commercial and rad-hard 
versions of devices reduces the process of upgrading to a flight ready board to a simple IC 
exchange with minor modifications to the timing parameters of the glue logic. As for 
parts showing reduced susceptibilities to SEUs, different technologies used to implement 
devices can have drastic differences in their SEU susceptibility, such as in the case of 
static or dynamic RAM. When these choices were available, the technology offering the 
lower susceptibility was used. 


£e Accelerator Testing 


One of the key functions of this system is its ability to test fault-tolerant software. 
In order to achieve that function, the user must be able to operate the system in a particle 
beam, which places added requirements on the implementation of the system. First, the 
system must be able to communicate with the host interface over distances of fifty feet, 
due to the shielding and physical layout of the testing facilities. Because heavy ion 
testing requires the system to be placed in a vacuum chamber, the device must be able to 
physically fit within the chamber and be reset remotely. The remote reset allows the user 


to recover a “hung” system without having to bring the chamber back to local pressure, 


reset the system, and then reapply the vacuum, which wastes expensive chamber/facility 


time. 
B. SYSTEM OVERVIEW 

The final block diagram for the TMR R3081 system as implemented is shown in 
Figure 3-1. Compared to the simulated system depicted in Figures 2-5 and 2-6, there are 


several differences. This section will discuss those differences. 
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Figure 3.1. TMR R3081 Block Diagram 
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The most important difference between the two designs is the addition of a 


System Controller to the implementation, since the Verilog® design suite acted as the 


System Controller during simulation. For the implementation, an FPGA was selected to 
perform the System Controller function. Its design is explained in Chapter IV. 

The other main design change 1s concerned with the I/O interface. Since the 
scope of the design in Ref. [9] did not require the loading and running of user programs, 
an interface to provide that functionality was not included. Additionally, the simulation 
used a System Interface module to read out the collected FIFO data. Although the System 
Interface was expected to be a laptop or similar system, its interface was simulated using 
a 104 bit wide interface, which could not be directly implemented into hardware. This 
left a wide latitude in the implementation of the I/O interface. For reasons discussed 
below, a dual-port interface was selected. 

Although this thesis focuses on hardware implementation, two other major efforts 
on this project were going on in parallel. Captain Kim Whitehouse, USMC, and 
Lieutenant Susan Greoning, USN, conducted research into the selection of an operating 
system for the TMR R3081 and the design of a ie Computer Interface (HCI). Their 
research garnered the selection of VxWorks as a real-time operating system and the 
design of a user interface for a software testbed which is presented in Ref. [12]. The 
important aspect of the close association of the hardware and software designers is the 
ability to effectively integrate the two designs, which is why several hardware design 
decisions are based on the software interface requirements. 

The functionality of the HCI plays a significant role in determining the design of 
the I/O interface. The HCI controls the TMR’s mode of operation, provides for a remote 


reset, enables the user to download and run programs, and also provides a path for data 
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collected in the FIFOs to be returned for analysis. Serious consideration was given to the 
use of a single communications port for this interface. Difficulties arose in the design 
when trying to implement the remote reset function with a single port. Because the CPUs 
control the transfer of data in a single port device, a “hung” system would not be able to 
reset because it could not respond to the reset signal at the port. In order to achieve this 
function a second port, controlled by the System Controller, was added to the 
implementation. 

With the decision made to use two ports, the next step was to determine what 
functionality was associated with each port. To help distinguish the two ports, which 
were depicted back in Figure 3.1, the processor-controlled port was named the Data Port 
and the System Controller controlled port was named the Control Port. Since the ability 
to download and run programs does not require interface with the System Controller, it 
was left on the Data Port. In order to free the processors from the overhead associated 
with transmitting the large FIFO data blocks and to provide the System Controller with 
direct access to control and resets, these functions were assigned to the Control] Port. 
This separation lets the user interact with the downloaded program through the Data Port 
and interact with the TMR hardware through the System Controller on the Control Port. 

Now that the major changes in the implementation from the simulation have been 
presented, the rest of the chapter discusses the individual pieces of the design. Section C 
presents the design and implementation of the system support elements that were not 


required during the simulation phase. The microprocessor and address latch selection is 


then addressed in Section D. The design of the Memory Space and its components 
follows in Section E. Finally, Section F presents the design of the FIFO buffer interface. 


G. SYSTEM SUPPORT ELEMENTS 


The detailed analysis of the TMR implementation starts with the system support 
elements. These portions of the system are critical to the systems functionality, setup, 
and testing, but are seldom included in simulations. One of the benefits of simulation is 
the ability to only include the required portions of the system. The implementation must 
account for all the elements that must be added to support the main system that were not 
included in the simulation, such as pull-up and pull-down resistors, decoupling 
capacitors, clocking and reset systems, mode select, and test connectors. 


1. Discrete Components 


One important change from a simulation environment to a _ hardware 
implementation is the inclusion of the discrete resistors and capacitors used in the 
circuits. When control signal lines are driven by devices that can tri-state their outputs 
pull-up or pull-down resistors are required to maintain the signal line at a known value. 
This prevents the device from misinterpreting the driver going into the high impedance 
state as an active signal. Additionally, decoupling capacitors are required to reduce the 
power fluctuations seen on the voltage bus during switching and to keep AC noise off the 
Debus. 


Pap Timing Interface 


The Timing Interface consists of two different clocking systems, one for the CPUs 


and one for the two universal asynchronous receiver transmitters (UARTs), which are 
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used to transmit data through the Data and Control Ports. The schematic for the Timing 
Interface is given in Figure 3.2. The two oscillators and the buffer/driver are all socketed 
to allow the user to easily change the oscillator frequency or the buffer/driver chip. The 


frequency selections and buffer/driver logic selections are discussed below. 
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Figure 3.2. Timing Interface 


a) CPU Oscillator 


For reasons discussed later, the 50 MHz model of the microprocessor was 
selected for this implementation. Ref. [13] requires the ClkIn signal, or CPU clock, to 
have a period between 40 ns and 50 ns, which translates to between 20 MHz and 25 
MHz. This is because the 50 MHz R3081 is restricted to the 1x Clock Mode, where the 
ClkIn signal is passed to an internal clock doubler and converted to a double frequency 
clock for signals internal to the processor. The CPU oscillator selected for this 
implementation was the 20 MHz model, which provided a slightly longer clock period to 


account for the added propagation delays of the Voter FPGAs. 
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b) UART Oscillator 


The UART Oscillator provides a reference input to the two UARTs that 1s 
used to generate the clock signals that control the serial communications between two 
terminals. For the two devices to communicate, their UARTs must be set up to transfer 
and receive data at the same rates. In order to accomplish this, the UARTs provide an 
internal register that can be loaded with a divisor to generate any baud rate equal to or 
lower than the reference signal. The divisor 1s usually used to generate one of the 
industry standard baud rates, such as 38.4, 57.6, or 115.2 kbps. This allows the user 
flexibility when selecting the interface frequency. 

The design goal was to provide the highest data transfer rate possible 
within the constraints of the I/O interface. Higher data transfer rates reduce the transfer 
time required for user programs and CPU FIFO contents. The original oscillator 
frequency selected was 16 MHz, which is the maximum allowed oscillator input to the 
UART. This frequency allows the UART to transmit at its maximum rate of 1Mbps. 
Unfortunately, other system constraints discussed in the I/O portion of the Memory Space 
section of this chapter could not support the maximum transfer rate. 

Since the maximum baud rate could not be supported, an attempt was 
made to maximize the number of standard baud rates the system could support. This led 
to the use of an 11.0592 MHz oscillator. This frequency allows for standard baud rate 
generation up to 115.2 kbaud, which is the maximum transfer rate currently supported by 


the HCI serial port. 
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Cc) Buffer/Driver Logic 


The main purpose of the Buffer/Driver chip is to buffer the low power 
oscillator signals into output signals strong enough to drive the CPU and UART clock 
inputs. The CPUCLK signal that emerges from the buffer/driver is the timing reference 
signal used by the CPUs. Since the input signal to the buffer from the oscillator is not 
referenced by any other component, the phase and delay of the buffered signal compared 
to the oscillator signal does not matter. The same applies to the UARTCLK, since its 
frequency only needs to be matched to the device it is communicating with. 

Most buffer/drivers come in packages of eight. In order to minimize the 
chip count, other uses of vacant glue logic chips were considered. In this case, three 
additional buffers could be used if the buffer/driver inverted its inputs. Two interrupt 
signals, UARTINT and TIMERINT, are generated by their respective devices in positive 
logic and the buffer/driver inverts them to the negative logic required by the CPUs. 
Additionally, the SYSCLK* signal generated by the CPUs as a reference signal for bus 
interface transactions is inverted by the buffer/driver to get the SYSCLK signal used by 
the memory controller state machines out of phase from the CPU reference clock. This 
allows the memory controller to meet the setup and hold times required by the CPU for 
read and write cycles without adding additional wait states beyond those required by 
memory access times. 

There is a very large pool of inverting buffer/drivers to choose from. The 
design goal used in the selection of the buffer/driver was a CMOS implementation with 


less than 12 ns propagation delay. The CMOS implementation reduces the static power 


oe) 
to 


dissipated by the device, which is also one of the major considerations when selecting 
parts for a space flight board. The restriction on the propagation delay was needed to 
ensure the Read Clock Enable and Acknowledge signals reached the CPUs soon enough 
to meet the required setup times. The Texas Instruments (TI) 74AHCT540 was selected 
because it met these requirements [Ref. 14]. Chapter IV will present a detailed analysis 
of these timing signals. 


3. Reset System 


Most computer systems have a single level reset circuit that 1s activated when 
power is turned on or by a push button switch. The TMR implementation uses a two 
level reset system that can be activated three different ways: power on, remotely, or by 
push button switch. The Reset System design evolved from the system requirement of 
being remotely reset and the use of serially programmed FPGAs. The main reason a two 
level reset was implemented was to allow the system to be reset without having the delay 
associated with reprogramming the FPGAs. A schematic diagram of the Reset System is 
shown in Figure 3.3. 

When power is initially applied to the system, the Supply-Voltage Supervisors 
(SVSs) sense the level of the supply voltage. When the supply voltage crosses a point 
near 3.6 V, their RESET outputs go active and the timer function is initiated, as long as 
the RESIN* input is not active. The timer function is determined by tg = 1.3 x 10°*C,, 


where ty 1s the delay time and C, is the total capacitance in the timer circuit. [Ref. 15] 
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Figure 3.3. TMR Reset System 


The first SVS, U72, uses a 7 UF capacitor in its timer cireuit to holdiiue 
BRD_RESET™* output low for 91 ms. The reset pulse allows the FPGAs to recognize the 
reset signal and setup for their initialization. The BRD_RESET™ signal is also routed 
back to one of the inputs of a 3-input AND gate. The AND gate combines the three input 
signals to the second SVS, U73. This feedback prevents the timer of the second SVS 
from starting until the first timer has finished. The FPGAs use the time delay provided by 
the second SVS, 1 second using a 77 UF capacitor, to load the serial caitiemaae data 
from their PROMs and to initialize. The one-second delay for loading clearly meets the 
200 Us pulse required by the processor [Ref. 10]. If the two step approach were not used, 
the processors would attempt to make memory accesses through the FPGA vote logic, but 
no signals would pass through the FPGA because all the inputs and outputs of the FPGA 
are in a high impedance state during initialization. 

After the power on reset (POR) is completed and the board is operating, there are 
two levels of reset the user can initiate. The first is a Board Level Reset. This reset goes 
through the complete POR routine. The second level of reset is a System Level Reset. It 


does not reset the FPGAs, but resets all the remaining devices in the system. 


Each level of reset has two methods of activation. The first method 1s through a 
normally open push button switch. Since the switch is normally open, the input to the 
AND gate used by the switch must be pulled-up to Vcc. When either switch 1s closed, an 
active low signal is applied to the respective AND gate, which causes the appropriate 
RESIN* input to go active. The second method is initiated remotely by the HCI. The 
HCI can send a new control word to the Control Register in the System Controller, which 
is described in Chapter IV. The System Controller drives the SCBRD_RST* and 
SCSYS_RST* inputs to the reset logic. When the System Controller activates one of 
these inputs, the appropriate level reset is initiated remotely. 


4. Mode Select 


The IDT R3081 processors have several features that are selected during system 
reset. The signals used to select these features are latched into the processor off the 
Interrupt bus by the rising edge of RESET*. The input signals and the functions they 


select are given on the following page in Table 3.1. [Ref. 10] 


Table 3.1. Mode Selectable Features. From Ref. [10] 









Since the Interrupt bus must provide both the Reset Vector and the Interrupt 
signals, a system to multiplex the two signals had to be designed. A tri-statable non- 


inverting buffer/driver was used to select between the two signals. The Interrupt bus is 
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routed through the buffer. During reset the buffer is held in the tri-state mode, effectively 
blocking the interrupt signals. This allows the signals selected by pull-up and pull-down 
resistors to be latched into the processor. After the reset is complete, the System 
Controller activates the INTEN* signal, which allows the interrupt signals to drive the 


processor interrupt inputs. A schematic of this system is shown in Figure 3.4. 
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Figure 3.4. Interrupt/Mode Logic 
The three synchronous interrupt inputs, SINT*{0..2], are hardwired to Vcc, since 
there was no need for the user to change them. They are hardwired to select Big 
Endianess, not Tri-stated, and Data Block Refill. Since VxWorks requires Big Endian 


functionality [Ref. 12], this mode selection was hardwired. Since there is no need to Tri- 
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State the processors outputs until another reset, essentially disabling the processor, this 
input was tied inactive. Additionally, the Data Block Refill signal was asserted to make 
the processor handle both data and address cache misses in exactly the same manner, as 
quad word reads. 

The remaining interrupt inputs, INT*[3..5], can be selected by the user at reset. 
This enables different functionality if the 50 MHz processors are swapped out for a 
slower frequency model. Since the 50 MHz R3081 is restricted to a one times clock input 
and half frequency bus operation, the Half-frequencyBus* and 1xClockEn* inputs, 
INT(3)* and INT(4)* are asserted during reset. Additionally, the Coherent DMA Enable 
is not asserted on INT(5)* because the implementation does not use DMA. 


5. Test Connectors 


Test connectors were designed into the schematic to allow for ease of 
troubleshooting. The initial design had connectors on each bus of the system. They were 
between each processor and the voter FPGAs, the voter FPGAs and the memory system, 
the FIFOs and the System Controller, and one collecting up all the control signals. 
Unfortunately, the Orcad tool used by the contractor to convert the schematics to a PCB 
Layout could not support the large number of pins in the system. The majority of the test 
connectors had to be pulled from the design. Only the connectors between the FIFOs and 
the System Controller remain in the implementation. For troubleshooting, chip clips will 


have to be used to extract the signals from the board. 
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D. MICROPROCESSOR AND ADDRESS LATCH 


Although the system requires the support elements to function correctly, the key 
element in the system is the microprocessor. In the case of the TMR system, it is actually 
the three parallel microprocessors. This section discusses the model selection of the 
R3081 microprocessor. Since the hardware is being presented as it is grouped in the 
schematics, the address latches are also discussed here. The schematic diagrams of the 
three microprocessor/address latch pairs along with the rest of the system are given in 
Appendix A. 


ie Microprocessor 


Even though the type of processor for the system was determined in Ref. [9], the 
frequency and version of the model for the implementation still had to be selected. The 
simulations conducted in the Ref. [10] used a 40 MHz system clock with a half-frequency 
bus. In order to have the ability to replicate the simulation values and push the design 
envelope, a processor with the ability to handle frequencies of 40 MHz or higher was 
needed. The 50 MHz IDT 79R3081-50MJ was selected because it has an allowable 
frequency range of 40 to 50 MHz. This processor comes in two versions, the base and 
extended. The only difference between the two is the extended version adds a full- 
featured Memory Management Unit (MMU) to manage the virtual to physical address 
translation. Since the simulation used a base version and the processor is socketed for 
ease of future upgrades, the base version was selected for this implementation and the 


extended version was left for future inclusion. [Ref. 10] 
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om Address Latch 


In order to reduce the pin count of the R3081, IDT chose to multiplex the address 
and data bus together. This generates a requirement for the designer to separate the two 
buses using a latch. The processor generates an ALE signal to capture the address into 
the latch. Normally, a transparent latch is used, which allows the address to appear on the 
address bus a short propagation delay after ALE goes high. This gives the memory 
system more time to access the desired memory location. 

In this design, the address latch that was simulated was a 74FCT373. The FCT 
family of logic uses fast CMOS as its core logic with TTL inputs and outputs. The high 
power consumption of the FCT logic family did not fit the design parameter of a future 
space flight board. This led to the selection of the 74AHCT573 for the implementation to 
take advantage of its low propagation delays and power requirements. [Ref. 17 and 18] 


E. MEMORY SPACE 


There are three types of devices found in the memory system of this design, 
Programmable Read Only Memory (PROM), Random Access Memory (RAM), and 
memory mapped I/O peripherals. This section will present an overview of the memory 
Space supported by the R3081 and the incorporation of the PROM, RAM, and I/O 
peripherals into the memory space of the implementation. 


1. R3081 Memory Space 


The R3081 supports a 4 GByte memory space, which is broken down into four 
distinct virtual address areas: KUSEG, KSEGO, KSEG1, and KSEG2. The KUSEG, or 


kernel/user segment, is a cacheable 2047 MByte area used for both kernel and user 
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processes. This gives the kernel access to user memory regions. KSEGO, or kernel 
segment zero, 1s a cacheable 512 MByte area that is always translated to the 512 MByte 
region of the physical address space beginning at address Ox0000_0000. Since this region 
is cacheable it is normally used for kernel executable code and some kernel data. 
KSEG1, or kernel segment one, is a non-cacheable 512 MByte area that is also mapped to 
the lowest 512 MByte area of physical memory. Since this section 1s non-cacheable, it is 
normally used for memory mapped I/O, boot ROM code, and operating system data areas. 
KSEG?2, or kernel segment two, is a cacheable 1023 MByte area analogous to KUSEG, 
but it is only accessible from kernel mode. This area is normally used by operating 
systems for stacks, process data, and dynamically allocated data areas. A diagram of the 
base version of the virtual to physical address translation is shown in Figure 3.5. [Ref. 10] 
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Figure 3.5. Virtual to Physical Address Translation. From Ref. [10] 





Zz. PROM 


Although the R3081 supports a 4 GByte memory space, there was no need to 


populate the entire space for this implementation. The driving factor for the PROM size 
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and type for the implementation was the operating system (O/S). Since a set of boot 
PROMs containing VxWorks was part of the O/S purchase, the PROM space was 
designed around the VxWorks PROMs, which are standard 27C0O10 PROMs with 512 
kBytes of storage and a 150 ns access time. This only required a minor change to the 
simulated PROM space from Ref. [9], which is the addition of two address lines to 
account for the increase from 128 kBytes to 512 kBytes. 

In order to have the ability of running user defined boot programs, a set of Atmel 
28C010-15 PROMs were ordered. The difference between the 28C0O10 and the 27CO10 
parts is that the 28s are electrically erasable, while the 27s need fifteen to twenty minutes 
under an ultra violet light to be erased, otherwise they are functionally equivalent. 
Because of long delivery times needed for the 28C0O10s, AMD 27C010-12s had to be 
substituted. These parts have a 120 ns access time versus the 150 ns access time of the 
VxWorks PROMs [Ref. 18]. Fortunately, the detailed timing diagrams presented in 
Chapter IV show that the memory controller state machine will support both 120 and 150 
ns accesses speeds without the need for changes. 

As far as their location within the 4 GByte memory space, the PROMs were 
physically located between addresses 0x1FCO_0000 and Ox1FC7_FFFF. The starting 
location was specified by the R3081 architecture, because 0x1 FCO_O000 is the address 
vector loaded into the Program Counter during a reset [Ref. 10]. The schematic of the 


PROM space can be found in Appendix A. 
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Si SRAM 


Since the PROM usually only maintains a small kernel of boot code, the processor 
needs a segment of SRAM to store the programs, applications and data that are needed 
during execution. The design of the SRAM space for the TMR system was based on two 
different factors: how large a memory space was needed and which segments of memory 
to populate. The following sections present the decisions leading to the final 
implementation of the SRAM space, which 1s shown 1n Appendix A. 

a) Size 

The memory space used tn the simulated model of Ref. [9] was a 128 
kByte block designed by placing four 32 k x 8 bit memory chips in parallel. This design 
proved more than sufficient for the simulations. In fact, only a small portion of the 
memory block had data placed tn it. Because of the success during simulation, the 
memory space for the implementation started with the same design concept. The only 
question that remained was how many of the 128 kByte blocks would be required. 

Because the Software Testbed function was expected to place a larger 
demand on the memory system, that model was used to determine the amount of SRAM 
needed for the implementation. In the Software Testbed model there are two contributing 
factors to the size of the memory: the O/S and user programs. The minimum SRAM 
requirement for a development system that includes all the standard VxWorks features ts 
2 MBytes [Ref. 19]. Because the implementation does not require the full complement of 
features, such as the networking capabilities, the O/S SRAM requirements are reduced to 


under 0.5 MByte. Allowing an additional 0.5 MByte for stacks and other dynamically 


allocated memory, the total memory requirement for the O/S is estimated to be 
approximately | MByte. Since the size of user programs can vary greatly, an additional 
1.5 MByte of SRAM was included in the memory space. This brought the total SRAM 
requirement to 2.5 MByte. 

Using the simulated model of 128 kByte memory blocks, it would take 20 
different memory blocks to implement the 2.5 MByte memory array. Furthermore, each 
memory block would require its own chip select signal and it would take 80 chips to 
implement, which would consume a very large area on the printed circuit board. In order 
to reduce these requirements, a 128 k x 8 bit memory chip was selected. This reduced the 
chip select signals to 5 and the chip count to 20. 

The IDT71024 CMOS Static RAM was selected for the implementation of 
the main memory. This chip was selected over other SRAM manufacturers because two 
other major components of the TMR system were also manufactured by IDT: the 
microprocessors and FIFOs. This was done to reduce the number of different 
manufacturers used and to increase interoperability. The initial specification called for a 
20 ns read/write cycle to allow for faster bus speeds when using 40 MHz processors. 
Because the 20 ns SRAM was not available in a timely manner, the IDT71024S12 was 
used, which has a faster read/write cycle time of 12 ns [Ref. 20]. 


b) Memory Segments 


As discussed in the beginning of the memory section, the R3081 supports 


four different virtual address segments. From the previous section, the design was 


43 


determined to consist of five different 0.5 MByte segments. This section looks at how the 
five SRAM segments are used to populate the four virtual address segments. 

Only two of the four virtual address segments are populated with SRAM. They 
are the KUSEG and KSEGO. Since KSEGO is used for kernel code, one segment of the 
O/S SRAM was placed there to provide a random access location for the contents of the 
PROMs to be transferred to and executed from. The remaining O/S segment and the 
three user segments were placed in the KUSEG in one contiguous memory space. This 
was done to allow the O/S greater flexibility when assigning code to memory locations. 

The other two virtual address segments, KSEG] and KSEG2, are not populated 
with SRAM in this implementation. Since KSEGI is uncacheable and used as the address 
area for the PROM and peripherals, there was no need to populate it with SRAM. 
Because KSEG2 1s restricted to kernel use and has the same functionality as KUSEG, 
KSEG2 was not populated in this implementation. This area can be populated in later 
designs simply by changing the Address Voter FPGA’s chip select equations, since the 
chip select signals are decoded from the voted upper address lines. 


4. Input/Output (I/O) 


Since the UART is a memory mapped peripheral, it seemed logical to discuss the 
design of the I/O interface in the Memory Space portion of this chapter. A peripheral is 
memory mapped if it is accessed by decoding an address placed on the bus. One of the 
design issues associated with the I/O system was selecting the address for its registers. 


Because the software development portion of this project had access to code mapping the 
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UART to OxBFEO_0003 and that address was not used by any other peripheral, the 
UART was assigned that address for sake of continuity. 

The I/O interface performs the important function of interfacing the TMR system 
to the HCI. Because the simulated — used the Verilog design suite to handle the 
interface, this portion of the system had to be designed from the ground up. The design 
of the interface consisted of selecting a type of interface and then implementing the 
selection. 


a) Interface Type 


There are many different types of computer system interfaces on the 
market today. When trying to determine which interface was best for this implementation 
of the TMR R3081, several factors were considered. They were the amount of data that 
needed to be transferred, how fast it needed to be transferred, and how far it needed to 
travel. This section looks at how these factors contributed to the selection of the interface 
for the TMR R3081 system. 

In this design there are two major uses of the I/O interface. They are the 
transfer of the user program to the TMR System from the HCI and the transfer of the 
FIFO daia to the HCI from the TMR System. Because radiation testing causes SEUs to 
be induced at a much higher rate than the space environment, the FIFO data transfer was 
considered to be the critical transfer. If the additional bus traffic before and after the 
interrupt is ignored and only the register writes to memory are read into the FIFOs, there 
is a requirement to transfer 140 register values from each processor [Ref. 10]. The 140 


registers are broken down by location in the Table 3.2. The 140 registers contain 816 
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bytes of data. In order to transfer each byte of data there 1s normally an overhead of three 
additional bits for start, stop and parity. This brings the total of bits to transfer to 8,976 
per interrupt for each processor. With three processors, the TMR system will need to 


transmit 26,928 bits to the HCI system for each vote error interrupt. 
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Table 3.2. R3081 Internal Registers 











The speed of the transfer is somewhat related to the size of the data 
element: the larger the data block to transfer, the higher you want the transfer rate. In this 
implementation the goal was to be able to transfer the FIFO data to the HCI in 
approximately 250 ms. This value was selected to allow four sets of interrupt data to be 
transferred per second. In order to transfer 26,928 bits in under 250 ms a minimum baud 
rate of 108 kbaud 1s required. 

One of the most critical factors in choosing the interface was the distance 
the data had to travel. One effect of increasing the transmission distance is a lowered 
maximum frequency of the driver. This happens because the long communication lines 
add capacitance to the load, which draw large currents and induce noise on the lines [Ref. 
21]. Due to the physical layout of the radiation chambers a transmission length of 
approximately fifteen meters 1s expected. This distance allows the testers to be shielded 


from the radiation beam during testing. 
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To summarize, the interface needed for the implementation should be able 
to transfer data near 108 kbaud over fifteen meters. Research into the different interfaces 
led to solutions ranging from an Electronic Industry Association (EJA) 232-F, also known 
as RS-232-F, to an Ethernet connection. In the end, the EIA-232-F was selected. Other 
than meeting all the criteria, this interface was selected because it was the easiest to 
understand and implement. This was important because the nature of this project has it 
passed on from student to student and anything that can reduce the amount of ramp-up 
time is a benefit. The actual implementation of the interface is presented in the following 
section. 


b) I/O Interface Implementation 


The I/O interface is implemented using a universal asynchronous receiver 
transmitter (UART) and a set of line drivers. The purpose of the UART is to convert 
parallel data from the TMR system to serial data for transmission or convert received 
serial data to parallel] data for use by the TMR system. It also generates the control 
signals used to establish communication between the two devices. The line drivers 
convert the serial data and control signals from digital values to signals capable of being 
transmitted across the serial cable. The selection of these two devices is presented in this 
section and their general interconnections are shown in Figure 3.6. The design 


schematics are located in Appendix A. 
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The selection of the UART turned out to be one of the easiest part 
selections involved in the implementation. The Texas Instruments TL16C750 was 
selected for the implementation for its maximum transfer rate of 1 Mbaud and its ability 
to meet the EJA-232 standards [Ref. 22]. Additionally, it had 64 byte FIFO buffers for 
both the transmitter and receiver, which allowed the processor more flexibility in reading 
and writing to the UART. It also has a Programmable Baud Rate Generator to allow 
different baud rates to be used for interfacing. Unfortunately, this part was not available 
without a lengthy delay, so the TL16C550C was substituted. This part was completely 
functional and pin compatible with the TL16C750, but only has 16 byte FIFO buffers. 
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Figure 3.6. Bus/UART/Line Driver Interconnections. From Ref. [22] 
Since the line drivers are the devices that adhere to the transmission 


standards, they became the most important aspect of the I/O interface. The initial set of 
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drivers selected was the Texas Instruments MAX3243. Because very few high end line 
drivers are available, it was very easy to make this part choice, but they were also 
unavailable for delivery. The Maxim MAX239 was then substituted in its place. This 
device meets all the requirements of the ELA-232-E standard and operates at baud rates of 
up to 120 kbps [Ref. 24]. This allows data to be transferred at the maximum allowed rate 
of the HCI at this time. 

Two important notes about the I/O interface are that if the HCI gets 
upgraded during follow on work to allow higher data transfer rates, the line drivers are 
socketed to allow for easy exchange. Also, the UART and line driver pair are 
implemented twice in the TMR R3081 system. One set is used for the Data Port and the 
other Set is used for the Control Port. 


F. FIFO INTERFACE 


The FIFO interface is used to collect data off the three microprocessors’ A/D 
busses during vote error interrupts. This is done by “snooping” the address and data 
information off the bus during the interrupt handler. The data contains each of the 
microprocessors’ internal register sets, which are transferred out the Control Port to the 
HCI for analysis. This section describes the design of the FIFO interface. A schematic 
diagram of the FIFO array is given in Appendix A. 


i. Control Bus Design 


The simulated system of Ref. [9] used two 1k x 18 bit FIFO buffers in parallel to 
collect the address, data, and control information from the busses of each processor. The 


data was then read out of the FIFOs using the System Controllers 104 bit wide System 
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Interface. The interface of the FIFOs and the UART, which replaced the System Interface 
in the implementation, created the design problem, because the 8 bit wide UART bus did 
not match up with the 18 bit wide FIFO bus. Two solutions for this problem came to 
mind: add glue logic between the FIFO and UART or redesign the interface. The later 
solution won out. By replacing the two Ik x 18 bit FIFOs with five 1k x 8 FIFOs the 
interface problem quickly disappeared. The first four FIFOs are used to collect the 
address and data information, while the fifth one collects all the control information. 
Since the FIFOs use tri-state outputs, all fifteen of the 8 bit wide FIFO output busses can 
be placed on the same bus as the 8 bit wide UART bus, which provides a streamlined 
connectivity to the UART. 


Me FIFO Selection 


For reasons discussed in the SRAM section, FIFOs manufactured by IDT were 
selected for the implementation. The large number of FIFOs to choose from allowed 
plenty of flexibility in the selection of the FIFO for the implementation. Since the 
simulated version of this design did not need to handle a large number of interrupts, the 
FIFO buffer could be relatively shallow. For the implementation, the design had to 
ensure enough space in the FIFO to prevent the FIFO from overflowing and losing data. 
By doubling the size of the buffer to a depth of 2,048 (2k), the system can support up to 
seven errors without overflow, assuming no data is transferred out. Because both the 
address and data bus contents are written into the FIFO for each register write. the 140 
registers require 280 positions in the FIFO for each interrupt. Unfortunately, the 2k FIFO 


was not available and the 4k FIFO had to be substituted in its place, which removed all 
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fears of overflow. The part number for this device is IDT72240LI5TP and its 
specifications can be found in Ref. [25]. 

This chapter has presented the design of the hardware used in the implementation 
of the TMR system. In that discussion three very important pieces were omitted: the 
design of the System Controller, Voters, and Memory Controller. One common element 
all three of these devices share is that they are designed in programmable logic devices, 
either FPGAs or PLDs. Because their design is more of a software implementation, they 


are presented together in the following chapter. 
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IV. PROGRAMMABLE LOGIC DESIGN 


Because programmable logic offers flexibility to the system designer, it was 
selected to perform several key functions in the TMR R3081 design. First, the Memory 
Controller and Memory Enable functions were implemented in Programmable Logic 
Devices (PLDs). The selection process used for the PLDs and a description of the signals 
they produce are presented in Section A of this chapter. Field programmable gate arrays 
(FPGAs) were used to implement the voting logic, address decoding, timer, and System 
Controller functions. The selection process and the design of the three FPGAs used in the 
system are presented in Section B of this chapter. Section C ends the chapter with a 
detailed timing analysis of the processor interface with all the penpheral devices that 
interact directly with the processors. The programmable logic components of the design 


are identified as shaded blocks in Figure 4.1. 
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Figure 4.1. Programmable Logic Device Identifier 
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A. PLD DESIGN 

The simulated system design presented in Ref. [9] used three user-defined 
modules to perform the Address Decoder, Memory Controller, and Memory Enable 
functions. Those designs were based on PLDs presented in Ref. [11], which is an 
Integrated Device Technology (IDT) system design example using an IDT R3051 
microprocessor. Although this is an older processor with fewer capabilities, the 
backward compatibility of the R3081 made it pertinent to this design. Because of its 
applicability, it was also used as a design template for the implementation designed in 
this thesis. 

The design example in Ref. [11] used three PAL22V10s to perform the Address 
Decoder, Memory Controller, and Memory Enable functions. Because of the number of 
inputs and outputs required for each of these functions, the smallest PLD that they would 
fitin was a PAL22V10. The FPGAs did offer one other option to the implementation of 
these three functions. Since the two Voter FPGAs generate the majonty of the input 
signals used by these PLDs, the functions could have been incorporated into one or both 
of the Voter FPGAs. It was decided to incorporate the Address Decoder into the Address 
Voter FPGA for the following reasons: 

e All the signals required to perform the address decoding were present in the 

Address Voter FPGA. 

e The FPGA had enough available I/O pins to accommodate the chip select 

signals. 

e The functions were similar enough that their consolidation did not obscure any 


integral functionality of the system. 
e The combination of the two functions streamlined the overall design. 
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Because the Address Decoder was incorporated into an FPGA, its design is presented 
later in the chapter. Although the Memory Controller and Memory Enable functions met 
most of the criteria used to incorporate the Address Decoder, they were left for 
implementation in PLDs. The main reason was to provide better visibility of their 
functionality for future changes or upgrades. 

The next step in the process was to select the PLD to be used in the design. Since 
the design example used 22V10s to implement their functions, it was also selected for the 
implementation in order to facilitate continuity. Because the 22V10 is a standard part that 
is fabricated by many companies, the specific manufacturer was not as important as the 
timing constraints of the available devices. The Atmel ATF22V10C-7PC was selected 
for use in the implementation. This electrically erasable CMOS device offered a 
maximum clock to output delay from registered outputs of only 2.5 ns and a maximum 
propagation delay of 7.5 ns [Ref. 26]. 

The next issue faced in the design of the PLDs was how to actually program them. 
Research led to a software program called WinCUPL, which is a Windows version of 
Universal Compiler for Programmable Logic (CUPL). Like other programmable logic 
compilers, CUPL defines its own language and syntax requirements for device design. 
Designs may be implemented as logic equations, truth tables, or state machines and then 
simulated using the software tools that are provided. The two most important factors 
leading to the selection of CUPL as the design environment was that it is produced by 
Logical Devices, Inc. (LDI) and that it has the ability to design FPGAs, Xilinx FPGAs in 


particular. The connection to LDI was important because the device programmer being 
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used is also distributed by LDI. This hardware/software combination provides good 
interoperability and is actually referenced in the CUPL Users Guide [Ref. 27]. For 
reasons discussed in the FPGA Design Section of this chapter, the ability to program 
Xilinx FPGAs was also thought to be important. 

The schematics showing how the Memory Controller and Memory Enable PLDs 
are interconnected within the TMR system are provided in Appendix A and the CUPL 
design and simulation files for the PLDs are provided in Appendix B. The next two 
subsections describe the two devices. 


1. Memory Controller PLD 


The Memory Controller PLD acts as the main piece of glue logic between the 
microprocessors and the memory space. When the microprocessors generate the signals 
initiating a bus transaction, the Data and Control Voter FPGA perform a majority vote on 
them and pass them on to the Memory Controller PLD. The Memory Controller uses 
those signals to generate the signals required by the memory space to perform the bus 
cycle and the signals the processor requires to terminate the bus cycle. This section 
describes the signals generated by the Memory Controller PLD. A diagram of the PLD 
with its inputs and outputs is given in Figure 4.2 and its program file is in Section 1| of 


Appendix B. 


SY SCLK 1 VOTNT- 
Navelcn D I1/CLK “peat 
AKT 3 |2 ‘Wal"4éa 
VERGS 4) 8 SUSERR 
LJARTICS” 5 I4 AEC) a's) 
EPHOMCS: 6 | YCEND 
VOTBURS 7 | 6 
VOTRD" 8 17 
VOTWH 9 18 

iC) hd f3 10 9 

ry C) ‘=32)2 11 110 

255 13 11 





12 


ATF22V 10C-5JC 
DIP.100/24/W.300/L1.175 


Figure 4.2. Memory Controller PLD 


a) Counter 


The Memory Controller PLD uses an internal four-bit counter to provide a 
timing reference for the different bus cycles. This allows the processors to interact with 
devices of various speeds by introducing wait states into the bus cycle for devices that can 
not respond in a single access time. Also, because the R3081 supports burst reads, which 
is a read of four consecutive memory locations in a single read cycle, the counter also 
provides a reference for each of the four memory accesses. 

The counter is implemented in the PLD by using pins I/O[7..10] for 
internal interconnect, which is why they are shown as unconnected in Figure 4.2. It uses 
the RESET*, CYCEND*, SYSCLK, VOTRD*, and VOTWR* inputs to determine when 
to count and when to reset back to zero. The RESET™® signal onginates from the System 
Reset signal and is used to reset the counter whenever a system reset is required. The 
CYCEND®* signal is another internally generated signal used to identify the end of the 
current bus cycle and to reset the counter back to zero. Its implementation is discussed 


next. The VOTRD* and VOTWR% signals are the majority voted RD* and WR signals 
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generated by the microprocessors to initiate the bus cycle. Unless the RESET* or 
CYCEND* inputs are asserted, the counter increments on each nsing edge of SYSCLK 
after the VOTRD* or VOTWR*% signal has been asserted. It will continue to count until 
either the CYCEND* or RESET™ inputs are asserted. 


b) Cycle End (CYCEND*) 


The CYCEND* signal is used to signal the end of the current bus cycle. 
Since all the signals generated by the Memory Controller and Enable PLDs are registered, 
they require a signal that will cause them to de-assert at the end of each bus cycle. This 
function could be performed by using the negation of the VOTRD* or VOTWR% signal, 
but it is not because they occur within the setup and hold times of the buffered/inverted 
SYSCLK* signal [Ref. 11]. 

The CYCEND* signal is generated from the RAMCS*, EPROMCS*, 
UARTCS*, TIMERCS*, INTCS*, VOTRD*, VOTWR*, VOTBURST™* inputs and the 
counter contents. The timing of the CYCEND®* signal is dependant on the latency 
required to access the device and the type of bus cycle that is being run. The five active 
low chip select signals (those that end in CS*) and the counter contents are used to 
determine the device latency. The VOTRD*, VOTWR*, and VOTBURSIT™ signals are 
used to determine the type of bus cycle. For this implementation, all the devices could 
perform a single word read or wnite within a single access time, except for the EPROM, 
which required one additional wait state for the read cycle and for which the wnte cvcle is 
not applicable. Quad word reads are only applicable to the SRAM and EPROM and are 


detected by the VOTBURST™* signal being asserted. During this type of read. the 
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CYCEND* signal is not asserted until the fourth word has been accessed, which takes six 
clock cycles for SRAM and seven clock cycles for EPROM. 


c) Read Buffer Clock Enable (RDCEN*) 

The RDCEN®* signal indicates to the microprocessors that the memory 
system has sufficient time to process the read request and will have valid data on the A/D 
bus when the microprocessor 1s ready to latch the data in. Because this signal has to be 
asserted prior to the nsing edge of SYSCLK™* for the data to be latched in on the next 
falling edge, valid data does not have to be on the bus when RDCEN*% is asserted, only 
prior to the next falling edge of SYSCLK*. During quad word reads, RDCEN* must be 
asserted and de-asserted four times. Along with enabling the latch process, the RDCEN* 
signal also serves to terminate the bus cycle, which will be explained with the ACK* 
signal. 

The RDCEN™* signal is generated by the five CS* signals, VOTRD*, 
VOTBURST™, and the counter. As the name implies, this signal is only asserted on read 
transactions. With exception of the EPROM, the remainder of the memory system can 
perform single word reads in a single clock cycle. Because of that, RDCEN* is asserted 
with the rising edge of SYSCLK while the counter contents is OxO. The EPROM requires 
one wait state, therefore, RDCEN™ is asserted on the following clock when the counter 
contents is Oxl. During the burst read cycles, RDCEN*% is asserted four times for each 
bus transaction. Because the Memory Controller uses SYSCLK as the reference for the 
State machine, the signal is asserted for one clock cycle and then de-asserted for the next 


one. Since the SRAM does not require any wait states, RDCEN* is asserted during the 
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clock cycles when the counter contents are 0x0, 0x2, 0x4, and 0x6. The EPROM requires 
a single wait state in the initial access time, but the remaining reads can be completed in 
the time required to toggle the RDCEN™* signal from on to off and back to on. Therefore, 
it is asserted when the counter contents are 0x1, 0x3, 0x5, and Ox7 for EPROM burst 
reads. 


d) Acknowledge (ACK*) 


The ACK*® signal is sent to the microprocessor to indicate that the memory 
system has processed the bus cycle sufficiently. The processor may then either move on 
to the next write buffer or release the internal core to process the read data. On both 
single word reads and burst reads, the microprocessor can implicitly generate ACK* and 
terminate the bus cycle based on the RDCEN*™* signal, although this technique will 
degrade performance on burst reads. To optimize performance, the ACK™* signal was 
generated by the Memory Controller for the burst read in this implementation. On writes, 
the microprocessor uses the ACK* signal generated by the Memory Controller to 
terminate the bus cycle. This specifies that the memory system has completed the 
transaction and that the processor can stop driving the A/D bus with the data. 

As with the other signals looked at, ACK* is generated by the five chip 
select signals, VOTRD*, VOTWR*, and VOTBURST™ inputs and the counter contents. 
The ACK®* signal for single word reads was not explicitly generated as discussed before. 
The burst read ACK*, which improved performance by releasing the processing core to 
use the data already in the read buffer, must not be generated more that four clock cycles 


prior to the end of the transaction. This ensures that the last word of the read is in the 
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buffer by the time the execution core expects it to be there. This signal is generated when 
the counter contents are 0x3 for SRAM reads and 0x4 for EPROM burst reads. This 
signal must be explicitly generated on all wntes. Since the EPROM can not be written to, 
all acknowledges for wnite transactions occur without wait states or when the counter 
contents are 0x0. 


e) Bus Error (BUSERR*) 


The BUSERR* signal is generated by the memory system to inform the 
microprocessors that they have tned to access an invalid address. The BUSERR® signal 
uses the counter to determine when this happens. The counter starts counting when a 
VOTRD* or VOTWR% signal is asserted by the processors to signify the start of a bus 
transaction. After initiating the bus cycle, the processor then waits for a RDCEN* or 
ACK™® signal from the memory system. These signals will never be generated, because 
they require a valid address to generate the chip select signal that they are dependant on. 
The counter will continue to count until it reaches OxF, which causes the BUSERR* 
signal to be asserted. 


fp Voter Interrupt (VOTINT*) 


The VOTINT™* signal is generated by the Memory Controller to indicate 
that a miss-compare has happened in one of the Voter FPGAs. This signal is routed to 
the INT3 input of the three microprocessors and held asserted until the processors 
acknowledge it during the interrupt handling routine. This signal uses the AVOTERR, 
CVOTERR, DVOTERR, and INTCS* inputs. The signal is asserted when any of the 


XVOTERR signals are asserted. When the interrupt handler performs a wnite to the 
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Interrupt Acknowledge Address, the INTCS* signal is generated and de-asserts the 
VOTINT™ signal. 


Ze Memory Enable PLD 


The Memory Enable PLD supports the Memory Control PLD during bus 
transactions. It does this by generating the read and wnite enable strobes for the memory 
system, read and wnite data enable signals to control the drivers on chips between the 
busses, and a positive and negative logic synchronous reset. The diagram of the Memory 
Enable PLD with its inputs and outputs is given in Figure 4.3 and its program file is in 


Section 3 of Appendix B. 
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Figure 4.3. Memory Enable PLD 


a) Read Enable (RDEN*) 


The RDEN® signal works in conjunction with the chip select signal, which 
is generated by the Address Voter FPGA, to activate the output drivers of a memory 
device. Because all the logic between the processors and the chip select signal is 
combinatonial, the possibility exists for the wrong chip-select signal to ‘glitch’ while the 


Address Bus is settling. In order to avoid the wrong address being accessed during this 
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time, the RDEN* signal is held high until later in the bus transaction. This allows the 
chip-select signal to stabilize and gives the memory device time to access the address 
location and move the data requested to its outputs. The RDEN* signal is then used to 
activate the device’s output drivers in order to drive the Data Bus. Unlike the write 
enable signals, only one RDEN®* signal is required. Since reads are always done as 
complete words, four bytes wide, there is no reason to select which bytes will participate 
in the read. The RDEN% signal is just passed to all four devices in the memory block in 
parallel. 

The RDEN% signal 1s generated using the SYS_RESET*, CYCEND*, and 
VOTRD* inputs. The RDEN* is asserted on the next nsing edge of SYSCLK after 
VOTRD* is asserted. The delay is used to allow the chip select signals to settle. The 
signal is de-asserted by CYCEND* being asserted. This turns the output drivers of the 
memory device off as early as possible in the bus transaction and helps reduce the 
possibility of bus contention. The SYS_RESET™* input only effects the RDEN* signal 
during resets, at which time it de-asserts the signal if it was asserted. 


b) Write Enables 


Since the R3081 supports byte, halfword, tri-byte and full word writes, the 
Write Enable strobes are used to determine which bytes of the word are going to be 
involved in the write transaction. The four write enable signals that are produced are 
WRENA*, WRENB*, WRENC*, and WREND*. WREND* corresponds to the most 


significant byte and WRENA* corresponds to the least significant byte. 
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The Wnite Enable strobes are generated by the SYS_RESET*, VOTWR%*, 
CYCEND*pVOTBEO*, VOTBE1*, VOTBE2*, and VOTBES* signals: The'GY@EnD*= 
and SYS_RESET™* signals are used in the same manner for the Wnite Enables as the 
RDEN* signal. Each of the Write Enable strobes corresponds to a single voted Byte 
Enable signal from the microprocessors. When the VOTWR* signal is asserted, the 
Wnite Enable strobe corresponding to each of the asserted Voted Byte Enable signals will 
also be asserted. Additionally, the R3081 supports byte, half word, tri-byte, and full word 


writes. To clarify the point, the truth table for this function 1s given in Table 4.1. 


Because the signals are negative logic, the Os signify an asserted signal. 


VOTWR* | VOTBEJ3..0] | WREND* | WRENC* | WRENB* | WRENA* | 
a 


l 





or 000020, St ee 


Table 4.1. Write Enable Assertion Table 


Cc) Data Enables 


The Read Data Enable (RDDATAEN) and White Data Enable 
(WRDATAEN*) signals are used to contro] the output drivers on devices separating 
different data busses in the system. In this system, the Data Voter FPGA separates the 
three microprocessor’s A/D Busses from the VDATA Bus and the driver/transceiver 
separates the VDATA Bus from the memory system’s DATA Bus. This has to be done to 


prevent bus contention when the processor turns the bus at the end of a read transaction 
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after a device with a slow turn off time has been accessed. The faster turn off time of the 
FPGA and driver/transceiver isolate the slower device until the bus is cleared for the next 
transaction. 

The WRDATAEN is a positive logic signal generated by SYS_RESET™, 
VOTWR%*, and CYCEND*. It allows data to flow from the processors, through the Data 
Voter, and into the memory system. As with the RDEN* signal, the SYS_RESET™ clears 
the signal on reset. The VOTWR%* is used to assert the signal and CYCEND* is used to 
de-assert it. By using the CYCEND®* to de-assert the strobe, the devices are tri-stated 
earlier in the cycle, which allows them more time to turn completely off before the 
processor drives the bus with the next address. The RDDATAEN® strobe works exactly 
the same as the WRDATAEN strobe, except it is a negative logic strobe and uses the 
VOTRD* signal instead of the VOTWR*. 


d) Synchronous Resets 


Since some of the peripherals in ihe Memory System required positive 
logic resets and some required negative logic resets, a reset signal had to be generated for 
both polarities. Although this could have been done by placing an inverter on the 
SYS_RESET™* signal to get the positive logic reset, it was implemented in the PLD. This 
solution provided a common location for both the positive and negative logic 
synchronous reset signals. These signals are just registered and registered/inverted copies 
of the SYS_RESET™ siemial. 

Although the Memory Controller and Memory Enable PLDs are key 


elements of the glue logic in this system, they are not the only programmable devices 
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used. The next section of this chapter presents the selection and design of the three 
FPGAs used in this system. 


B. FPGA DESIGN 


Although the simulated system in Ref. [9] used three user-defined modules to 
perform the voter functions and the Verilog Design Suite to perform the System 
Controller function, they were implemented in three FPGAs in this design. This section 
of Chapter IV presents the FPGA selection process and the design of the three FPGAs 
used in the system. Schematics of the FPGA designs are presented in Appendix C. 


Ly FPGA Selection 


Unlike the PLD selection, the FPGA selection process required more research. 
The initial focus of the FPGA search was trying to find an FPGA vendor that 
manufactured the same FPGAs in both commercial and rad-hard versions. This was 
important in maintaining the design goal of implementing a system that could easily be 
converted for future space use. If the FPGA selected for the implementation was 
available in both commercial and rad-hard versions, the upgrade to a space flight model 
could be completed easily. The programming files designed for the commercial model 
could also be used to program the rad-hard FPGA. This search criterion narrowed the 
possible vendors down to Actel and Xilinx. 

The next factor used to narrow down the search was FPGA size. Since FPGAs 
are a very powerful implementation, the limiting factor for this design was not based on 
internal logic, but on the number of user definable I/O pins. The FPGA had to be large 


enough to hold a complete functional unit. That is, a complete 32-bit voter needed to fit 
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in a single FPGA. This required a minimum of 130 user definable I/O pins: three 32-bit 
inputs, one 32-bit output, and at least two control lines. Actel had one rad-hard FPGA 
that met the requirements with 140 user definable /O pins and Xilinx had three with 192, 
288, and 384 user definable I/O pins. 

Although any one of the four FPGAs listed in the previous paragraph would have 
worked in the implementation, a selection had to be made for the implementation. The 
Xilinx FPGAs offered one advantage over the Actel FPGAs in the way that they were 
programmed. Since the programming data is actually fused into the Actel FPGAs, they 
would have to be physically taken off the PCB and reprogrammed whenever changes in 
the design had to be made. The Xilinx FPGAs use a serial PROM (SPROM) to store the 
programming data. Whenever the system is reset, the program data is restored into the 
FPGA. By using the Xilinx FPGAs in the design phase, the 20-pin SPROM is removed 
for reprogramming, not the 240-pin FPGA. Although this may not sound like much of an 
advantage, the large number, small feature size, and close spacing of the FPGA pins 
would make its removal and replacement a very difficult task. This narrowed the viable 
options down to the Xilinx FPGAs. 

The remainder of the selection process came down to how the implementation 
could best be packed in the FPGAs. The total pin count for the entire FPGA solution was 
383 YO pins. This would have actually fit in the 384 pin FPGA. This option was not 
taken because it did not leave enough room for design changes and future upgrades. The 
first design used the 384 pin FPGA to perform the voting function and the 192 pin FPGA 


to perform the System Controller function. When minimum order quantity requirements 
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and cost were factored in, the final design used three 192 pin FPGAs. The System 
Controller, Address Voter and Decoder, and Data and Control Voter were each 
implemented in separate FPGAs. 

Once the FPGA selection process was completed, the next consideration was how 
to program them. The software tool used to program the PLDs, WinCUPL, does offer a 
method for writing, compiling, and converting designs for use with Xilinx FPGAs. The 
only drawbacks were that the designs had to be entered in a hardware description 
language (HDL) and the conversion process was rather complicated. Because of this, 
other options were looked into for the FPGA design. It turned out that the new text book 
being used for the logic design course at NPS, Ref. [28], came with a student edition of 
the Xilinx Foundation software. Xilinx designed this software for the purpose of 
programming Xilinx FPGAs. After determining that it would support the FPGA selected 
for the implementation, it was selected as the design medium for the FPGAs. This 
software allows the design to be implemented in HDL, State Machine Design, Schematic 
Representation, or any combination thereof [Ref. 29]. 

Since the majority of the voter functions are combinatorial, the decision was made 
to design the Address and Data Voter FPGAs using the schematic representation. Their 
designs are presented in the next two sections. Although the actual implementation of the 
System Controller was not completed, its design is presented in Subsection 4 of this 


chapter. 
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Dre Address Voter, Memory Decoder, and Timer FPGA 


The first FPGA implemented was the Address Voter FPGA. Its functions are 
performing the majority vote on the three 30-bit address busses, decoding the upper 
thirteen bits of the voted address lines into chip select signals, and providing a system 
timer. The logic design of these functions is presented in this section and their schematic 
diagrams are provided in Section 1 of Appendix C. 


a) Address Voter 


Since the R3081 multiplexes its data and address pins, demultiplexers 
must be used to separate the Data and Address busses. The A/D bus of the 
microprocessor is split into two different paths, one registered and one not. The 
unregistered path is the Data Bus and the registered path is the Address Bus. When the 
R3081 initiates a bus cycle, the A/D bus is driven with the address to be accessed. The 
R3081 then asserts the Address Latch Enable (ALE) strobe and the transparent latches of 
the demultiplexer drive the Address Bus with the address to be accessed. The negation of 
ALE latches the address off the A/D bus into the demultiplexer, where it is held on the 
Address Bus until the following bus cycle when ALE is asserted again. Since all three of 
the microprocessors do this in parallel, the three CPU Address Busses must be majority 
voted prior to being passed to the Memory System. 

The vote function for the Address Bus was designed hierarchically using 
Macros, which is a term Xilinx uses to describe the building blocks or sub-functions used 
to create an overarching design. First, a 3-bit Majority Voter/Error Detector, which is 


presented in Figure 4.4, was designed. This module takes three inputs and performs a 
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majority vote on them and passes the output to the zero input of a multiplexer. The 
DATAA input is directly connected to the other input of the multiplexer. The Force input 
is connected to the control input of the multiplexer. When Force is not asserted, the 
multiplexer passes the voted data to the circuit output. When Force is asserted, the 
multiplexer passes the DATAA input to the circuit output. The circuit also uses a three 
input AND and a three input NOR, which are ORed together to generate a negative logic 


error signal when any of the input bits differ from the other two. 
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Figure 4.4. 3-Bit Majonty Voter/Error Detector 
The next step was to combine four of the 3-bit Majonty Voter/Error 
Detectors into a macro called VOTER4. The schematic for the VOTER4 macro is given 
in Appendix C. This macro takes three four-bit wide busses and a Force input and 
produces two four-bit wide busses, a majority voted output bus and an error bus. The 
Force input causes the data on the DATAA Bus to be forced onto the OUT Bus through 


the multiplexer, which bypasses the vote logic. 
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Because the R3081 uses the four Byte Enable signals to select which of the 
four bytes are participating in the bus cycle, the Address Voter only needs vote the 
remaining 30 bits of the bus, because the Byte Enable signals are voted by the Control 
Voter. The Address Voter takes the three 30-bit Address Busses and a Force input to 
generate a 30-bit Voted Address Bus and a 30-bit Error Bus. It was implemented by 
placing eight of the four-bit wide voters in parallel, the two most significant bits are not 
used. The 30-bit Error Bus had to be reduced down to a single output, which was done 
by ORing the signals together. Since the Error signals are active iow and the output is 
active high, the operation turned out to be a logical NAND. The Address Vote Error 
(AVOTERR) signal was then passed to the appropriate output pin. The Address Voter 
also separated the Voted Address Bus into two sections, the lower seventeen bits and the 
upper thirteen bits. The lower seventeen bits are sent to the output pins to make the 
VOTADDR[2..18] Bus, which is passed on to the Memory System for decoding. The 
upper thirteen bits are passed on to the Memory Decoder, which is presented in the 
following section. 


b) Memory Decoder 


The Memory Decoder takes the upper thirteen bits of the Voted Address 
Bus from the Address Voter and decodes them into chip select signals. In order to 
prevent multiple addresses being mapped to the same memory location, a full address 
decoding scheme was implemented, which means all thirteen of the upper address bits 
were used to generate the chip select signals. A chip select signal is generated for each of 


the five SRAM memory blocks, the EPROM, the UART, the Timer, and the Interrupt 
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Acknowledge. Table 4.2 shows the chip select name, its physical address, its virtual 
address, and the memory segment it belongs to. The addresses correspond to the thirteen 
bits of VOTADDR[32..19] with three zero bits concatenated to the end. The table shows 
that the Kernel SRAM resides in KSEGI, the remainder of the SRAM resides in the 
KUSEG, and all of the penpherals and the EPROM reside in the uncached KSEG1 as 
discussed in Chapter I. The Memory Decoder schematic is presented in Section 1 of 


Appendix C. 


Signal Name Peripheral Physical Virtual Address Memory 
Address Segment 


URAMOCS* mas ver KUSEG 
URAMICS* 
EPROMCS* KSEG! 


UARTCS* UART Ox 1FEO OxBFEO KSEG1 


Table 4.2. Chip Select Memory Map 




















Cc) Timer 


The implementation of the timer in this design turned out to be one of the 
most time consuming problems faced. The initial plan was to create the timer using an 
integrated circuit designed specifically as a timing device. Unfortunately, neither of the 
two timers selected could be found, because both had become obsolete. Because 
VxWorks requires a system timer to function properly [Ref. 19], the timer had to be 


implemented in one of the FPGAs. 


The sole purpose of the timer is to interrupt the processor at predetermined 
time intervals. In response to the interrupt, the operating system will update the system 
clock. In order to prevent the system from spending too much time processing timer 
interrupts, the goal was to keep the number of interrupts under 100 per second, but as 
close to a whole number as possible. 

Since SYSCLK is used by the timer and 1s operating at 10 MHz, 40 
interrupts per second would require the counter to count to 250,000 before asserting the 
interrupt. Since 250,000 is closer to 2'® than 2!”, the design was easier to implement with 
a 19-bit counter. By using 2'® ANDed with 2'° as the reset and interrupt signal, the 
counter will count to 263, 168 and then assert the interrupt. This works out to 38 
interrupts per second. 

To provide this functionality, a 19-bit counter macro was designed out of 
Toggle flip-flops. The basic counter was designed by running the SYSCLK signal to the 
first flip-flop’s clock input. The output of the flip-flop was inverted and tied to the next 
flip-flop’s clock input. This was done repeatedly until nineteen flip-flops were in series. 
The Timer schematic is provided in Section 1 of Appendix C. 

Although this provided the basis for a timer circuit, certain functionality 
had to be added to make it useful to the system, such as a Timer Enable, Timer Disable, 
Timer Interrupt Enable, Timer Interrupt Disable, and a Timer Interrupt Acknowledge. 
The system allows the Timer Mode to be changed by performing a dummy write to a 


specific set of addresses. Table 4.3 lists the Timer Mode and its associated address. 


iis 






0x04400000 


Table 4.3. Memory Mapped Timer Modes 








This functionality was designed into the system using the 
VOTADDR[17..2] Bus, the TIMERCS* signal, three D flip-flops, and some 
combinatorial logic. The sixteen bits of VOTADDR[17..2] are fully decoded to provide 
five signals representing each of the desired functions: TIMERINTACK, TIMEREN, 
TIMERDIS, TIMERINTEN, and TIMERINTDIS. Figure 4.5 provides a schematic of the 


Timer functionality design. 
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Figure 4.5. Timer Enable/Disable/Interrupt Design 
The TIMEREN signal is ANDed with an inverted TIMERCS* signal and 
routed to the D input of the Timer Enable flip-flop. The TIMERDIS signal is ANDed 
with the inverted TIMERCS* and routed to the reset input of the Timer Enable flip-flop. 


The output of this flip-flop is routed to all of the Clock Enable and Toggle Enable inputs 
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of the T flip-flops. Whenever the Timer Enable flip flop is set, the Timer will count. If 
the flip-flop is reset, the Timer will sit idle. To set the Timer Enable flip-flop, a wnte 
transaction is performed to address 0x04400004. This asserts the TIMERCS* and 
TIMEREN signals, which puts a one on the input to the Timer Enable flip-flop. The next 
nsing edge of SYSCLK latches the data into the flip-flop and enables the timer. The 
output of the flip-flop is fed back to the input to hold the enable after the bus cycle has 
ended. The same method is used for the Timer Disable, except the signal is routed to the 
flip-flop’s reset input and 1s asserted when a wnite is done to address 0x04400008. 

When the counter reaches 263,168, the reset and interrupt signal is 
asserted. It is routed to the D input of the Interrupt flip-flop and the reset inputs of all the 
T flip-flops. On the next nsing edge of SYSCLK, the counter is reset to zero and the 
Interrupt flip-flop is set. When the processor acknowledges the interrupt by wniting to 
address 0x04400000, the inverted INTCS* is ANDed with the TIMERINTACK signal 
and routed to the reset input of the Interrupt flip-flop. The next nsing edge of SYSCLK 
resets the Interrupt flip-flop and de-asserts the interrupt. 

The output of the Interrupt flip-flop is ANDed with the output of the 
Interrupt Enable flip-flop. If the Interrupt Enable flip-flop is not set, the zero it supplies 
to the AND gate prevents the interrupt signal from being passed out of the FPGA. The 
Timer Interrupt Enable flip-flop works exactly like the Timer Enable flip-flop, except the 
TIMERINTEN and TIMERINTDIS signals are used when wnites are done to addresses 


0x04400010 and 0x04400018, respectively. 


3: Control and Data Voter FPGA 


Although the Control and Data Voter FPGA was implemented second, it was 
actually the easier FPGA to design. The main reason for that was that the Voter/Error 
Detector was already designed and just needed to be imported into this design. The only 
new designs needed were the Bidirectional Transceiver part of the Data Voter and the 
Synchronization Signal. The design of the Control Voter, Data Voter/Transceiver, and 
Synchronization Signal are presented in Subsections a, b, and c of this section, 
respectively. The schematic design of the Control and Data Voter is provided in Section 
2 of Appendix C. 


a) Control Voter 


The R3081 has eight outputs that act as control signals in this design: four 
Byte Enable signals, Burst (BURST*), Read (RD*), Write (WR*), and Data Enable 
(DATAEN*). When these signals leave their respective processors, they are combined 
into two 4-bit busses, BECPUx{[3..0] and CTRLCPUx[3..0]. The small x in the name 
represents the specific CPU the signals come from, such as A, B, or C. As an example, 
CTRLCPUA[0..3] is the BURST*, RD*, WR*, and DATAEN® signals from CPU A 
bussed together, where BURST™* is bit zero, RD* is bit one, WR* is bit two, and 
DATAEN* is bit three. 

The Control! Voter design uses two of the VOTER4 macros in parallel. 
The first macro takes the three CTRLCPUx[3..0] busses and the Force signal as inputs to 
generate the VOTCTRL[3..0] and CVOTERR{3..0] buses. This is done by performing a 


majority vote and comparison on the three input busses and passing the appropriate 
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output based on the Force signal, as described in the Address Voter presentation. The 
second macro generates the VOTBEN[3..0] and BEVOTERR[3..0] busses in the same 
manner using the three BECPUx[3..0] busses and Force signal. The VOTCTRL and 
VOTBEN busses are then routed to their respective output pins. The two 4-bit busses of 
voter error signals are individually ANDed together to form the CVOTERR signal. This 
signal 1s combined with the Synchronization Signal, which is discussed later in this 
section, prior to being routed to its output pin, where it is used by the Memory Control 
PLD to signal a miss compare. 


b) Data Voter/Transceiver 


The Data Voter/Transceiver is designed the same as the Address Voter 
with two exceptions: it uses a different macro to perform the voter function and it is 
designed to be bidirectional. For the voter function, a new macro named VOTER8 was 
designed. Although the macro redesign was not required, it was done to help the 
readability of the top-level design. The VOTER8 HT is designed in the same manner 
as the VOTER4 macro except eight 3-bit Voter/Error Detectors are used in parallel 
instead of only four, which gives it an 8-bit wide path instead of the 4-bit wide path of the 
VOTER4 macro. This reduced the number of macros needed to vote a 32-bit bus to four. 
The four 8-bit error busses generated by the four VOTER8 macros in the Data 
Voter/Transceiver are combined in the same manner as the error busses in the Address 
Voter to form the DVOTERR signal, which is also combined with the Synchronization 
Signal discussed in the following section. The combined signal is then passed on to its 


respective output pin. 
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Because the information passed across the data bus must flow in both 
directions, the Data Voter/Transceiver had to be designed for bidirectional data flow. The 
three data busses leaving the processors on write cycles need to be combined through the 
vote logic and passed on to the Memory System. On reads, the single data bus coming 
from the Memory System must bypass the vote logic and be split into three copies that are 
passed back out to the processor's A/D busses. This was done using multiple paths 
separated by tri-state buffers. The RDDATAEN* and WRDATAEN signals from the 
Memory Controller PLD are used to control the enable pins of the tri-state buffers. 
Figure 4.6 shows how the transceiver portion of the Data Voter/Transceiver was designed 


for each of the thirty-two bits on the data bus. 





Figure 4.6. Transceiver Logic Design 


The Transceiver logic is designed to flow through and around the voter 
logic. On write bus cycles, data flows from left to mght across the diagram, with the three 


input pins being driven by their respective processors. The Memory Control PLD will 
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assert the WRDATAEN signal, which enables the BUFE tri-state buffers. The data flows 
through each of the processor input line tn-state buffers, the vote logic, the output tri-state 
buffer, and to the output pin. The voted data also returns towards the input pins, but is 
stopped by the disabled BUFT tn-state buffers, which are controlled by the 
RDDATAEN* signal. 

On read bus cycles, data flows from nght to left across the diagram, with 
the single input pin being dnven by the Memory System. As discussed earlier, the 
Memory Control PLD asserts the RDDATAEN* signal and de-asserts the WRDATAEN 
signal on read cycles. The signal on the input pin is routed through the input buffer and 
to the three output BUFT tri-state buffers. The asserted RDDATAEN*% signal enables the 
tri-state buffers and lets the data flow through to the processor side pins. The de-asserted 
WRDATAEN*® signal prevents the read data from flowing back into the vote logic. 


c) Synchronization Signal (SYNC) 


Because skew will exist between the processors, the information on the 
Address, Control, and Data busses will not arrive at the voters at exactly the same time. 
Because of the way the Vote Error signal is designed, it will detect these differences and 
will be caste whenever the three signals do not match, including when the bus is 
transitioning and not expected to match. The purpose of the Synchronization signal is to 
prevent these ‘false’ voter error assertions from leaving the Voter FPGAs. This was done 
by ANDing the outputs of the Voter Error signals with the Synchronization signal, which 


will prevent the error signal from leaving the FPGA unless SYNC is asserted. 
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The SYNC signal was designed to be asserted during both read and wnite 
cycles while valid data was expected to be on the Address, Control, and Data busses. 
Since the address being used for the current bus cycle is captured by the demultiplexer, 
the Address bus is held constant once ALE has been de-asserted. As for the Control bus, 
its value is asserted early in the bus cycle and maintained throughout. The Data bus was 
the key to the SYNC signal. On reads, RDCEN* is used to indicate valid data is on the 
bus. On writes, the processor drives the bus shortly after ALE is de-asserted. The ACK* 
signal is then used to terminate the bus cycle. Since both the RDCEN* and ACK* 
signals indicate there is valid data on the bus, they were used to generate the SYNC 
signal, along with the VOTRD* and VOTWR*® signals. For the read portion of the 
SYNC signal, VOTRD* is ANDed with RDCEN*. For the write portion, VOTWR% is 
ANDed with ACK*. These two signals are then ORed together. Since these signals are 
active low, the AND-OR is logically an OR-AND, which is shown in Figure 4.7. The 
combined signal is then inverted and ANDed _— an inverted SYSCLK, which is 
accomplished logically with a NOR gate. The combination with the inverted SYSCLK 
holds SYNC from being asserted until the second half of the clock cycle, which gives the 
data bus ample time to settle after the memory system drives the bus on wnite cycles. The 


SYNC signal is then routed to its output pin. 
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Figure 4.7. SYNC Signal Design 


4. System Controller FPGA 


As discussed before, LT Payne used the Verilog Design Suite to act as the System 
Controller in the simulated design that was presented in Ref. [9]. The hardware 
implementation chose to design the System Controller in an FPGA, since they are one of 
the few programmable devices that are robust enough to hold all the functionality 
required of the System Controller. As the design progressed and the test plan started to 
take shape, the priority assigned to the design of the System Controller began to decrease. 
Although no testing was conducted during this phase of the project, the initial plan for 
testing the system is expected to progress as follows: 

e Test and analyze system support signals to ensure the system comes out of 

power-on-reset correctly. 

e Test system in single microprocessor mode and validate correct operation of 

each processor individually. 

e Test TMR mode of operation, without collecting bus data. 

Because the System Controller is not required to perform the first two sets of tests and the 
other two Voter FPGAs and the PLDs are, the priority was placed on the design of the 


other programmable devices. This section presents the initial design of the System 


Controller, although its implementation was left for follow-on work. 
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The System Controller serves several functions in the TMR system. It enables the 
interrupts for the microprocessors after resets, initializes the Control UART, collects and 
transmits FIFO data, and controls the mode that the TMR R3081 system functions in. 
The following subsections describe the requirements for each of the System Controller 
functions and the initial design considerations in implementing them. Since most of the 
functions are independent and translate easily to finite state machines (FSM), it is 
recommended that future work use the Xilinx Foundation software to implement the 
System Controller, since FSM design is one of the design alternatives provided by the 
software. 


a) Interrupt Enable (INTEN*) 


The INTEN* signal is used by the Mode Select hardware to multiplex the 
INT*[3..5] bus of the microprocessors. While the INTEN* signal is not asserted, the 
74AHCT541 buffer’s outputs are tri-stated and the INT*[3..5] bus is driven by pull- 
up/pull-down resistors, which are used to select the mode the processors operate in. 
Since the mode inputs are latched in by the RESET™ signal’s nsing edge, the INTEN* 
signal should be asserted after RESET* is negated and before the microprocessor’s 
interrupts are enabled. 

The only two signals required in the generation of the INTEN* signal are 
SYSCLK and RESET*. RESET* should be tied to the D input of a flip-flop and 
SYSCLK to the clock input. The Q output of the flip-flop should then be inverted to 
generate the negative logic INTEN* signal. This design generates the INTEN® signal on 


the first rising edge of SYSCLK after RESET™ is negated. Since the generation of the 
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INTEN* signal does not rely on any other function within the System Controller, this 
function can and should happen simultaneously with the other functions. A diagram of 


the INTEN* function is provided in Figure 4.8. 





Figure 4.8. INTEN* Signal Design 


b) Control UART (CUART) Initialization 


Since the majority of the System Controller functions revolve around the 
collection and transmission of data between the TMR system and the HCI, the CUART 
plays a very large supporting role to the System Controller. Neither the FIFO Data 
Transfer nor the TMR Mode Control functions can happen before the CUART 1s 
initialized. Therefore, it is very important that this function take place as early in the 
system initialization process as possible. 

The CUART initialization is the process of configuring the UART to 
communicate with both the CTRLDATA[0..7] bus and the device on the opposite end of 
the serial port. The minimum initialization requirements for the CUART are to setup the 
communications channel (baud rate, parity, even/odd, number of start and stop bits) and 
to select the operation mode (polled or interrupt). If the interrupt mode is selected, the 


initialization should also enable the interrupts. This is done through access to the 


83 


CUART Control Registers, which can be wnitten to and read from. The specific values 

used in the control registers for the initialization of the UART are developed in Ref. [23]. 
The System Controller should be able to read from and wnite to the UART 

in order to accomplish this function. The state machine should be designed to follow this 


pattern for a write cycle: 


e Assert the CUART chip select signal (CUARTCS%*), 

e Assert the Control Register address on CTRLADDR[2..4], 
e Assert CUARTADS* (From Address Voter), 

e Assert data on CTRLDATA[0..7], 

e Assert Read Enable signal (CURDEN?*), 

e Negate CURDEN*® (Latches data into register), 

e De-assert CTRLDATA[0..7] and CUARTCS*. 


The state machine should also be able to produce this pattern of signals for a read cycle: 
e Assert the CUART chip select signal (CUARTCS*), 
e Assert the Control Register address on CTRLADDR[2..4], 
e Assert CUARTADS* (From Address Voter), 
e Assert Write Enable signal (CUWREN*), 
e Read data off CTRLDATA[0..7], 
e Negate CUWREN* and CUARTCS*. 
In order to accomplish the read and write operations, all the signals required for the read 


and write functions were routed between the System Controller FPGA and the CUART. 


Cc) FIFO Data Collection 


After the CUART is initialized, the System Controller must collect the 
FIFO data before it can transfer it to the HCI. The final concept of data collection 
involves a FIFO Array that continuously collects data. The initialization of the FIFO 
Array would have the Wnite Pointer ten addresses ahead of the Read Pointer. The Read 


and Write Pointers are pointers inside the FIFO that determine which location is read 
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from and wnitten to. While normal execution of instructions occur, the FIFOs perform 
two read/‘“dummy wnite” pairs for each bus cycle. It is referred to as a dummy write 
because the output enables are not asserted and it only serves to increment the Read 
Pointer. The first read/“dummy wnite” collects the address information off the A/D 
busses and discards the address information from five bus cycles previous. The second 
read/““dummy write” serves the same function, only for the data portion of the bus cycle. 
This keeps five bus transactions in the FIFO Array at all times. When a VOTINI™ is 
detected, the “dummy wnite” 1s discontinued while the bus information is collected and is 
restarted after all the FIFO data has been transferred to the HCI. This scheme provides 
the five bus cycles prior to the error and the internal register contents of the processors to 
the HCI for analysis. 

Because of the large scale of the project, the approach taken with this 
function of the System Controller was to design a function that would collect the A/D bus 
information during the interrupt handler. This fulfilled the minimum requirement of 
providing the contents of all the internal registers of the three processors to the HCI for 
analysis. Once the system is running and has been tested, this function can easily be 
expanded to provide the additional five bus cycles. 

For the minimum functionality, the System Controller only requires five 
signals, VOTINT*, SYSCLK, VOTWR*, FWRCLKO, and INTCS*. The System 
Controller monitors VOTINT* for indication of a voter error. When VOTINT™* is 
asserted, the System Controller uses the write enable strobe (FWRCLKO) of the FIFO to 


write both the address and data information from the A/D bus into the FIFOs. The 
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FWRCLKO signal is generated by combining the SYSCLK and VOTWR* signals. 
SYSCLK is ANDed with an inverted VOTWR*. Since all the write bus cycles are two 
clock periods long and the data is latched into the FIFO on the nsing edge of the write 
clock, the first rising edge of FWRCLKO latches the address and the second rising edge 
latches the data. Since this signal should not be active at all times it should have an 
enable that is based on the VOTINI™ signal. This can easily be implemented by latching 
VOTINI™ into a D flip-flop and using its inverted output to enable the FWRCLKO signal. 
After the interrupt handler has finished storing all the processor registers to memory, it 
should acknowledge the interrupt, which negates the VOTINT™* signal. With the 
VOTINT negated, the FWRCLKO signal will be disabled and data collection will cease. 


A diagram of the FWRCLKO signal design is shown in Figure 4.9. 
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Figure 4.9. FWRCLKO Signal Design 
Since the processors complete the interrupt handler and re-enable their 
interrupts before the System Controller can transfer all the FIFO data to the HCI, the 
System Controller must be able to determine if an additional interrupt has occurred while 


data was being transferred from the previous one. Implementing an up/down counter in 
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the System Controller can do this. The Data Collection system will provide the input for 
the up counter and the Data Transfer system will provide the input for the down counter. 
When the INTCS* signal is asserted and negated, the counter will increment by one to 
indicate a Vote Error Interrupt has been serviced. The fact that the counter contents is not 
equal to zero is the signal for the Data Transfer system to begin transferming data to the 
HCI. 


d) FIFO Data Transfer 


The FIFO Data Transfer system has two functions. First it transfers a 
Header byte to the HCI to indicate which FIFO’s data is about to be transferred. It then 
transfers the complete contents of that FIFO to the HCI. Because the data transfer is a 
cyclical operation, the best way to implement it is by using state machines. This section 
will explain the Header byte and the state machine designs. 

(1) Header Byte Design. When the HCI starts receiving data 
from the TMR system, it has to know what the data represents in order to rebuild the data 
structure and to conduct analysis on in. This information comes from two locations, the 
interrupt handler and the Header Byte. By knowing the instructions the interrupt handler 
is performing and the order it is performing them, the HCI can determine exactly what 
each byte stored in the FIFOs represent. The list may look something like: R1, R2, ...., 
Kj PC. StatussCause.s 

The Header also provides information regarding the rebuilding of 
the data structure. The Header contains two elements of information, which processor the 


data onginated from and which FIFO in the array the byte corresponds to. The three 
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processors in the TMR system are referred to as Processor A, B, and C. The fifteen 
FIFOs in the system are broken down into arrays of five assigned to each processor, 
which are numbered zero through four. It takes two bits of the Header byte to identify the 
three processors and three bits to identify the FIFO. The breakdown of the header word is 
presented in Figure 4.10. Bits 0 and 1 make up the Processor Field, bits 2, 3, and 4 make 


up the FIFO Field and bits 5, 6, and 7 are reserved for future use. 


Bit Bit Bit Bit Bit Bit Bit Bit 
i 6 5 4 3 2 1 0 


Reserved For FIFO Processor 
Future Use 000 —0 100 ~4 00O-~A 
001-1 101-NU O1-B 
O10=2 iden JOS C 
Ole 3 111-NU  11-Not Used 
NU = Not Used 








Figure 4.10. Header Byte Design 

For each of the three processors, the data from one bus cycle is 
split into ten bytes with two bytes stored in each FIFO. The first byte corresponds to the 
address portion of the bus cycle and the second byte corresponds to the data portion. 
When the Data Transfer system transfers the data to the HCI, it will transfer all of the data 
corresponding to the current vote error from one FIFO before moving on to the next 
FIFO. This method of transferring the data was preferred because it reduces the latency 
of the transfer by not having to switch between FIFOs after each byte. The state 
machines of the FIFO Data Transfer system, which are explained next, coordinate the 


movement of the data from the FIFOs to the UART for transmission to the HCI. 
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(2) State Machine Design. Because the majonty of the 
elements of the FIFO Data Transfer system involve sequential tasks, the system is best 
suited to being designed as a set of finite state machines (FSMs). The FIFO Data 
Transfer system is designed as a hierarchical set of five FSMs: Data Transfer FSM, 
Processor Select FSM, FIFO Select FSM, Byte Select FSM, and Data Wnte FSM. This 
section presents their design and discusses their interactions. 

The Data Transfer FSM is the uppermost FSM in the system. It 
monitors the Interrupt Counter discussed in the FIFO Data Collection section. When the 
counter is no longer equal to zero, the Data Transfer FSM changes from the Wait state to 
the Transfer Data state. While in this state, the FSM asserts an enable for the Processor 
Select FSM and waits for the Transfer Complete signal to be returned, which signals the 
completion of the FIFO data transfer. When the Transfer Data operation is completed, 
the FSM returns to the Wait state and checks the value of the Interrupt Counter, which the 
Processor Select FSM decrements. If the counter is zero, the FSM will idle in the Wait 
state. If the counter is still not equal to zero, the FSM returns to the Transfer Data state. 


The Data Transfer FSM is given in Figure 4.11. 
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Figure 4.11. Data Transfer FSM Design 
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The next FSM is the Processor Select FSM. The purpose of this 
FSM is to sequence through the three processors’ FIFO arrays during the data transfer. 
This FSM waits in its initial state until enabled by the Data Transfer FSM. When 
enabled, the FSM transitions to the Processor A state, where it asserts an enable for the 
FIFO Select FSM and waits for a CPU Complete signal. The CPU Complete signal 
indicates that the FIFO contents of the current processor have been transferred to the HCI. 
The FSM then completes this operation for Processors B and C. After Processor C’s 
FIFO contents are transferred, the next state decrements the Interrupt Counter and asserts 
the Transfer Complete signal. The FSM then transitions back to its Wait state. The 


Processor Select FSM is shown in Figure 4.12. 






Wait Processor A) {Processor B Processor C 
Transfer 


ProcSel—En |CPU Complefe| CPU CompletelCPU Comalete Complete 





Figure 4.12. Processor Select FSM 


The next FSM in the hierarchy as the FirO Seleet Fort: ie 
purpose of this FSM 1s to sequence through each of the five FIFOs in the FIFO array 
during a data transfer. This FSM waits in its initial state until enabled by the Processor 
Select FSM. When enabled, the FSM transitions to the FIFO 0 state. which corresponds 


to the most significant byte (MSB) of the A/D FIFOs. Durng the FIFO 0 state, the FSM 
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enables the Wnite Data state machine and waits for a FIFO Complete signal. The FIFO 
Complete signal indicates that the Wnte Data FSM has transferred both the Header and 
the contents of the current FIFO to the UART. When the FIFO Complete signal is 
received, the FSM transitions to the next state and the process is repeated for each FIFO 
in the array. The sequence for the FIFOs is A/D MSB, A/D next to MSB, A/D next to 
LSB, A/D LSB, and Control byte. After the Control FIFO has been transferred, the next 
state in the FSM asserts the CPU Complete signal and then transitions to its Wait state. 


The FIFO Select FSM is shown in Figure 4.13. 
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Figure 4.13. FIFO Select FSM Design 
The Byte Select FSM is the next FSM in the hierarchy. The 


purpose of this FSM is to sequence through the Header and each of the bytes of data in 


9] 


the FIFO during a data transfer. This issue was complicated by the fact that the code to 
the O/S was not available. In order to build a Rapid Prototype of the HCI that could be 
tested, the initial data block agreed upon between the hardware and software system 
designers was 41 of the processor’s internal registers. The 41-register block contains the 
10 CPO control registers and 31 of the 32 general-purpose integer registers. RO was not 
transferred because it is a constant zero. Since the bus is sampled twice for each write 
cycle, the System Controller is expected to put 82 bytes of data in the FIFO during the 
interrupt handler. When the Header is added to this, the Byte Select FSM has 83 bytes of 
data to sequence through and Wait and FIFO Complete states. This brings the initial total 
to 85 states for this FSM. It waits in its initial state until enabled by the FIFO Select 
FSM. When enabled, the FSM transitions to the Header state, which asserts the Data 
Wnite Enable and waits for the Byte Complete signal. The Byte Complete signal 
indicates that the Write Data FSM has transferred the Header byte to the UART. When 
the Byte Complete signal is received, the FSM pandncns to the next state and the process 
is repeated for each byte in the FIFO. After the last byte in the FIFO has been transferred, 
the next state in the FSM asserts the FIFO Complete signal and then transitions to its 
Wait state. Since the Byte Select FSM 1s so large, only the initial portion of the design is 


shown Figure 4.14. 
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Figure 4.14. Byte Select FSM Design 





Finally, the last FSM in the FIFO Data Transfer function is the 
Write Data FSM. This state machine is responsible for issuing the control signals 
required to transfer data from the FPGA or FIFO to UART. This FSM was descnibed in 
the CUART Initialization section. After the selected byte is latched into the UART, the 
Wnite Data FSM asserts the Byte Complete signal and returms to its Wait state, which 
wraps up the design of the FIFO Data Transfer system. 


é) TMR Mode Control 


The last functionality to present in the System Controller is the TMR 
Mode Control. The TMR Mode Control serves two purposes, it allows the user to 
remotely reset the TMR system and to select between the TMR and Single Processor 
Modes of operation. This section presents the design of the TMR Mode Control. 
Although the TMR Mode Control performs two functions, it can be implemented in a 
single 8-bit register with the outputs tied to the signals required to initiate the required 
operations. Bit O is used to select between the TMR and Single Processor Mode. Its 
output is the FORCE signal. When this bit is a zero, the system operates in the TMR 
Mode. When it is a one, the system operates in the Single Processor Mode. Bit 1 


initiates a system-level reset by routing the inverted output of the register to the 
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SYS_RESET output of the FPGA. Bit 2 initiates a board-level reset by routing the 
inverted output of the register to the BRD_RESET output of the FPGA. A diagram of the 


TMR Mode Select Register 1s given in Figure 4.15. 


Bit | Bit | Bit Bit | Bit | Bit Bit Bit 
7 6 5 ~ 5 7 1 0 
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Figure 4.15. TMR Mode Select Register Design 


w. DETAILED SYSTEM TIMING ANALYSIS 


With all the hardware specified and the programmable devices designed, the next 
Step in the process is to conduct a detailed timing analysis of the system. The purpose of 
the timing analysis is to determine if the system operates in the manner expected and to 
ensure that no bus contentions exist. A bus contention occurs when multiple devices are 
driving the same lines at the same time, which could lead to a destructive fault. 

Since the processors can conduct three different types of bus transactions, the 
timing analysis was performed on each type of bus transaction and is presented in the 
same fashion: Single Word Read Cycles, Quad Word Read Cycles, and Single Word 
Wnite Cycles. Additionally, each device that the processors can interact with for a 
particular bus cycle had to be analyzed. The following sections present the results of the 


timing analysis. A timing diagram has been generated using Timing Designer 
= =) oS = ro) = 
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Professional, a software tool developed by Chronology. Like transactions have been 


combined where possible. 


1. Single Word Read Cycles 


Single word reads can be characterized by three phases: the processor initiates the 
bus cycle, the Memory System responds to the request, and the processor terminates the 
bus cycle. When the timing analysis was conducted for all the devices, it was found that 
the SRAM, UART, Timer, and Interrupt all have the same timing diagram. Only the 
EPROM had a different timing analysis for single word read cycles. 

The processors initiate the bus cycle by asserting the appropniate control signals 
on a nsing edge of SYSCLK*. In this case, the RD* signal is asserted, A/D bus is en 
with the address, and ALE is asserted. BURST™ is not asserted for single word reads. 
The assertion of ALE causes the latches in the A/D demultiplexer to become transparent 
and allow the address information to continue to the Address Voter FPGA. Near the next 
falling edge of SYSCLK*, ALE is negated and the processors turn the A/D busses to 
receive the data supplied by the Memory System. The negation of ALE latches the 
address into the transparent latches. The processors will wait in this state for the data 
indefinitely, unless the BUSERR* signal is asserted. 

The address busses are voted and decoded by the Address Voter FPGA and passed 
on to the Memory Systems address lines and chip selects. The Control Voter also votes 
all the Control lines during this time, which supplies the signals required by the Memory 
System to respond to the bus cycle. The assertion of VOTRD*, without VOTBURST™*, 


signals the Memory Control PLD that a single word read cycle has been initiated. 
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Regardless of which chip select signal is asserted, the Memory Control PLD asserts the 
RDEN* and RDDATAEN* signals. This enables the output drivers of the device being 
read from and directs the data to flow from the Memory System towards the processors in 
the bus transceivers and Data Voter FPGA. The asserted chip select signal determines 
how many wait States are generated by the Memory Control PLD. The EPROM was the 
only device that required a single wait state, all the other devices could conduct zero wait 
state reads. For the zero wait state devices, RDCEN* was asserted at the same time as 
the RDEN* signal and one clock cycle later for the EPROM, which signaled the 
processors that the Memory System had finished responding to the bus cycle. 

The assertion of RDCEN* by the Memory Controller PLD must occur prior to the 
rising edge of SYSCLK*. This requirement is easily attained because SYSCLK is used 
to drive the PLDs state changes and it is very nearly exactly out of phase with SYSCLK* 
due to the very small propagation delay of the inverting buffer. The processors then latch 
the data off the A/D buses on the next falling edge of SYSCLK*, along with negating all 
the control signals asserted for the bus cycle. The zero wait state read takes two 
SYSCLK* cycles to complete and the single wait state read takes three clock cycles to 
complete. The timing diagrams for the zero and single wait state timing analysis are 
presented in Figures 4.16 and 4.17, repectively. The diagram indicate that no bus 


contentions exist on any of the busses. 
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Figure 4.16. Zero Wait State Read Timing Diagram 
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In Quad-Word Read Cycles 


In order to reduce the latency involved in setting up read operations, the R308] 
supports quad-word reads. The bus cycle is initiated in the same fashion as a single word 
read, except the processors also assert the BURST™* signal. When the Memory System 
responds with the RDCEN* strobe, the processors latch the data off the bus and 
increment the CPUADDR[3..2] bus from 00 to 01. The Memory System then responds 
with another RDCEN* strobe and the process is repeated four times, which reads the 
addresses corresponding to 00, 01, 10, and 11 on CPUADDR[3..2]. After the fourth 
RDCEN*, the processors will terminate the read cycle. In order to provide better 
response, the R3081 lets the designer apply the ACK* signal up to four clocks before the 
last data word is latched in. This lets the processor core start processing the words that 
have already been latched into the read buffer, as long as the last word is in the buffer by 
the time the processor requires it. 

When the timing analysis was conducted, it was determined that the only two 
devices capable of performing quaded reads are the SRAM and EPROM. The other 
devices do not have linear memory spaces or multiple registers that require successive 
writes. When the SRAM’s and EPROM’s timing diagrams were compared, they were 
very similar. As expected, the SRAM was able to conduct zero wait state reads for each 
of the four reads. What was unexpected, was that the EPROM was able to conduct zero 
wait state reads for the last three reads after an initial single wait state latency dunng the 
first read operation. Because the SRAM can conduct four single-word reads or a single 


quad-word read in eight cycles, there is no savings in doing this. The EPROM 1s able to 
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reduce the number of cycles required to read four words during the quad-word read due to 
the added delay of waiting for the Memory Control PLD to toggle the RDCEN*® signal. 
This adds a complete clock cycle while the RDCEN® is taken high for the EPROM 
address to be accessed. The EPROM quad-word read saves a total of three clock cycles 
over four single-word reads. Because the timing diagrams for the quad-word reads were 
so long, they were split into two parts for legibility. The timing diagram for the SRAM 
Zero Wait State Quaded Word Read is given in Figures 4.18 and 4.19. The timing 
diagram for the EPROM Single Wait State Quaded Word Read is given in Figures 4.20 
and 4.21. As with the single-word read cycles, the quad-word read timing diagrams 


indicate that no bus contentions exist. 
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Figure 4.18. Zero Wait State Quaded Word Read Timing Diagram (1* Half) 


ye 


40s | 50ons 600s 700s 
| I | | 





SysCIKN 


SysClk  \\ ri \A y/ \A [if x\ y/ 
| RON Neg 
RON/BursiN 











L/ 
VolRON/VatBursiN {/ 
ne a i 
VotAda [3.2] DS EE : 
~» Hold Time <-—» Hold Tir 
| OE to Valid ial OE to Valid —e 00 
CPU ee >) 
a. i Looe" 
VData Ss 
pO — 
& 
Data ee 
ae SEE aa 
: Rate Che 
ROENN/R dD ataEnN Vy, 
State Change State Change State Change tee 
ROCEnN YA L/ \A i 
State Change 
ACKN 
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3: Write Cycles 


The R3081 also conducts write cycles in three phases, which are very similar to 
the phases of the read cycle. First, the processors initiate the bus cycle by asserting the 
WR%* control line, ALE, and the A/D bus with the address. After the processors negate 
ALE, they drive the busses with data instead of Ciel for the Memory System to 
respond. After the processors have initiated the bus cycle, the Memory Controller 
responds to the VOTWR* signal by asserting the WRDATAEN and ACK*® signals. The 
WRDATAEN signal sets the transceivers and Data Voter FPGA up to allow data to flow 


from the processors to the Memory System. The ACK* signal is the signal the processors 
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are waiting for that indicates the Memory System has had sufficient time to process the 
bus transaction. The processors then terminate the bus cycle. 

Because EPROM is not wnitten to, no timing analysis was required for this type of 
bus cycle. The remaining devices were all able to perform zero wait state writes. Their 
timing diagram is provided in Figure 4.22. As with the other timing diagrams, there are 
no bus contentions during the write cycle either. 

Although the detailed timing analysis brings together the hardware and software 
designs and checks them for errors, nothing substitutes for performing tests on the actual 
circuit boards to determine if they were designed correctly. The next chapter presents the 
steps taken to turn the implementation design into an actual circuit board with integrated 


circuits, switches, capacitors, and connectors. 
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V. MANUFACTURING AND DESIGN REVIEW 


Chapters II through IV presented a simulated design of the TMR R3081 system 
and the hardware and programmable logic designs required in the implementation of the 
design. This chapter presents what was done with those designs in order to produce a 
printed circuit board (PCB) for use in follow-on research and testing. 


A. PCB FABRICATION 


One of the issues not discussed in the earlier chapters was the selection of a 
software suite that could be used to draw the TMR system schematics, which ties in with 
the PCB fabrication. The initial inclination was to use the Verilog Design Suite, since it 
had been used for the simulated design. Unfortunately, the system had been recently 
upgraded and all the bugs had not been worked out. After spending the better part of six 
weeks trying to get the Verilog solution to work, it became apparent that it was not going 
to be a viable option. When the search turned to PC based software, two schematic 
capture software tools emerged as the front eS OrCAD and Protel. After using both 
of the software’s demonstration programs and talking to the contractor that was going to 
perform the PCB layout, the OrCAD software was chosen over the Protel. The two main 
reasons the OrCAD software was selected was that it offered better parts libraries and it is 
the same software that the contractor uses. Since the libraries could be checked pnor to 
Starting the design, all the parts required in the TMR system were available in the OrCAD 
software. Additionally, since the contractor also uses OrCAD, the designs could be 
imported directly into the PCB layout portion of the OrCAD suite, which also saved time 


in the manufacturing process. 
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Along with the OrCAD schematics, the contractor was given a list of 
requirements concerning the layout of the system. There could be no traces under the 
processors, the CPUCLK traces should be as close to the same length as possible, and the 
IC chips used on the PCB should be socketed. The restriction on the traces under the 
processors comes from Heavy Ion Testing. The traces could interfere with the test by 
generating additional 1ons when impacted by the test beam. The new particles could then 
cause errors in the processor, which would skew the test data. The CPUCLK traces need 
to be as close to the same length as possible to reduce the skew between the signals. This 
stipulation was included because the synchronization of the three processors was thought 
to be the most critical aspect of the design. If there was too much skew between the 
signals, the processors might not finish a bus cycle without a miss compare. Finally, the 
socketed parts allow for quick and simple part exchanges for the programmable devices 
or for upgrades to the other devices. 

Once the OrCAD schematics were delivered to the contractor, he created a Net 
List out of the project. The Net List was then used to create the PCB layout. Before 
sending the board out to the fabricators, the contractor attempted to order the parts 
specified in the schematics. It was during this time that the problem with the delivery 
time on many of the parts selected in the design was discovered. The design was then 
updated to reflect the changes that needed to be made because of the lack of part 
availability and the layout was sent to the fabricators. When it returned, the contractor 


soldered in all the sockets and the parts that were not put in sockets pnor to returning the 
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finished board. Figure 5.1 shows a picture of the TMR R3081 PCB with the functional 


blocks identified. 
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Figure 5.1. TMR R3081 PCB 


WHITE WIRES 


B. 


Although the hardware design was well thought out prior to sending it to the 


g, the PLD and FPGA designs had not been completed. This 


was done to streamline the design process in order to have a working prototype as early as 
possible. Having the early prototype would allow the software development team time to 
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contractor for manufacturin 


integrate the O/S and HCI into the design. Unfortunately, the plan did not work as 
expected and the design of the PLDs and FPGAs revealed additional routings that needed 
to be made, which are called White Wires. This section presents the design changes that 
were discovered during the PLD and FPGA designs, which are presented in Table 5.1 for 


ease of reference. 


Pin 184 Pin 4 

Ein 25 ine 

Pin 23 Pin 14 

Pin 14 Voter Pin 185 
Jencey sls) Voter Pin 186 
Pin 11 Voter Pin 171 
Pin 22 Voter Pin 172 
Pin 21 Voter Pin 173 
Fine23 Pin 144 


Table 5.1. Table of White Wires and Cuts 




















When the Timer system was redesigned because the parts were no longer 
manufactured, the TIMERCS* signal was removed from the design. It was discovered 
during the Memory Controller PLD design that the “dummy wnites” to the Timer required 
a chip select signal to be generated so the Memory System would generate the proper 
signals to acknowledge the write, even though nothing was actually being wnitten to the 


timer. 
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Another issue discovered during the Memory Controller PLD design required the 
moving of the RESET™ signal. The original PLD design used pin 13 of the PLD as part 
of the Wait State Counter. Since pin 13 is an input only pin, it could not be used for the 
feedback options that were required of the counter. The easiest way to remedy this 
problem was to move the RESET™* signal input from pin 14 to pin 13 of the Memory 
Controller PLD and cut the trace leading to pin 14. 

During the detailed timing analysis, a bus contention was discovered on the 
processors’ A/D busses. The VOTRD* and VOTWR* signals were being used to control 
the direction of data flow through the Data Voter/Transceiver. Because they were driven 
so early in the bus cycle, the Data Voter/Transceiver would drive the A/D buses while the 
processors were still driving them with the address values. In order to prevent the bus 
contention, the RDDATAEN* and WRDATAEN signals were supplied to the Data 
Voter/Transceiver to perform the direction control function. Since these signals are 
synchronized to SYSCLK, they provide an additional half cycle of delay before the 
busses are driven, which prevents the bus contention. 

During the design of the SYNC signal in the Data and Control Voter, it was 
discovered that the contro] signals supplied to the FPGA were not adequate to generate 
the SYNC signal. When the timing requirements presented in Chapter IV were analyzed, 
it was determined that the SYSCLK, RDCEN*, and ACK* signals were required by the 
Data and Control Voter FPGA to generate the SYNC signal. 

The last White Wire needed was discovered during the design of the System 


Controller FPGA, which was onginally going to use the SYS_RESET™* signal to signal 
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the assertion of the INTEN* signal. Because the processors are using the synchronous 
RESET™ signal for their reset function, this signal also had to be added to the System 
Controller FPGA for the INTEN* signal. If the SYS_RESET™* signal was used, the 
INTEN* signal would be asserted too early, which could cause the processors to initialize 
in the wrong mode. 

Once the White Wires and cuts are added to the TMR R3081 system, it will be 
ready to start the process of testing. The design of the PLDs, FPGAs, and the detailed 
timing analysis were conducted in conjunction with LT Damen Hofheinz, USN, who will 
be conducting the testing and follow on research. This was done to provide as seamless a 
transition as possible. The following chapter concludes this portion of the project and 


discusses possible areas for follow on research. 
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VI. CONCLUSIONS AND FOLLOW-ON RESEARCH 


The previous chapters have introduced the concept, provided background 
matenal, and presented the work completed on this project to date. This chapter will 
present the conclusions drawn from the project and possible areas for follow-on research. 


A. CONCLUSIONS 


The high prices and declining availability of radiation hardened devices has 
caused system designers to turn to using COTS devices in radiation environments. 
Although the COTS devices offer reduced design-to-orbit time, lower cost, and use of 
cutting edge technology in the design, they are also much more susceptible to SEUs. 

The TMR system implemented during this research provides a tool to reduce the 
risk posed by the SEUs in two different ways: by testing software algonthms or by 
masking out the fault. First, the TMR Software Testbed allows testing of fault-tolerant 
software in a radiation environment without having the expense of placing a test case in 
orbit. This allows the reactions of the software to be monitored when an SEU is 
encountered. When the TMR system is used as a hybrid fault-tolerant computer system, 
the hardware will mask out the error, reset the faulty processor to match the state of the 
other two processors, and continue processing from the point the error occurred. 

During this portion of the project, the simulated design model was expanded, 
upgraded, and implemented. The system was expanded by the addition of the support 
elements that were not required in the simulation phase. The addition of the System 
Controller to the design and its requirement to handle the data transfers between the HCI 


and the TMR System improve the processors availability for user programs. During the 
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implementation, the integrated circuit chips in the system, with exception of the 
microprocessor, were selected, the schematic designs were drawn, and the programmable 
logic devices were designed. Finally, a detailed timing analysis of the complete system 
design was performed, during which it was determined that there were no bus contentions 
and all the devices’ setup and hold times were met during all modes of operation. 

The simulated model of this system from Ref [9] and the hardware 
implementation presented in this research both prove the TMR concept to be a valid 
method of dealing with the effects of SEUs on COTS devices. This research advanced 
the project by implementing the system in hardware, which required the design of the 
System support elements, /O space, memory space, FIFO interface, and the System 
Controller. Although the System Controller implementation was not completed and no 
developmental testing was conducted, the detailed timing analysis presented in Chapter 
IV supports the claim that the design will work as intended. 


B. FOLLOW-ON RESEARCH 


Because of the relatively short time students get to actually conduct research, 
projects as large as this one are not normally completed by one student. The projects are 
segmented and each student in the progression will advance the project. This section 
explains some of the areas where follow-on research can be conducted, such as 
completion of the TMR Testbed, Radiation Testing, conversion to a space flight board, or 


applying the TMR design to a state-of-the-art (SOTA) processor. 
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iB Completion of TMR Implementation 


This thesis presented a partially functional TMR R3081 system. The hardware 
implementation, along with the programming for the Memory Controller PLDs and the 
Voter FPGAs allow the system to function as a stand alone computer system with the 
added function of detecting and correcting single bit errors occurring in any of the 
processors. In order to provide the full capability of the system, the System Controller 
FPGA needs to be programmed and operationally tested. This will allow the system to 
store and transmit data collected from each of the processors during a Voter Interrupt to 
the HCI for analysis. This will also complete the hardware design phase for this system. 

Once the hardware is completed, the system will be ready for the software 
integration, which consists of two parts, the O/S and the HCI, which are detailed in Ref. 
[12]. The VxWorks O/S is installed on the hardware by loading it onto the EPROMs. It 
then needs to be tested to ensure the device drivers and the interrupt handlers are 
configured correctly. The HCI is incorporated into the overall system by connecting its 
Data and Control Ports to the Data and Control Ports of the TMR system, since they are 
both stand-alone systems that communicate via two serial cables. The HCI and hardware 
need to be tested to ensure the communication paths are operating correctly and the data 
format being transmitted by the hardware matches the format expected by the HCI. When 
these items are completed, the system will be a fully functional TMR system that is able 
to detect and correct single errors in any of the processors and provide the data 
corresponding to that error to the HCI for analysis. The system is then ready for radiation 


testing and modifications for other uses. 


latel 


Ds Radiation Testing 


When the hardware and software integration is completed, the TMR R3081 
system will be ready for its operational testing. Because this system is designed to detect 
and correct radiation induced SEUs, its final operational testing must be done in the 
presence of radiation. Regional test facilities include the particle accelerators at the 
University of California, Davis and University of California, Berkeley. These facilities 
generate beams of high-energy particles that radiate devices placed in their path. By 
placing one of the processors in the radiation beam, SEUs can be induced in the device 
and the reaction of the system can be captured and analyzed by the HCI. 

The data captured by the HCI during the radiation testing can be used for two 
different purposes. First, it will validate the concept of a hybrid TMR fault tolerant 
system. Previous versions of TMR processor systems rebooted the system when an SEU 
was detected in one of the processors. This system uses the two valid processors, the 
Voters, and the memory system to reconfigure the faulty processor to match the two valid 
processors. This technique saves all the time and data that is lost when the recovery plan 
from an SEU is acomplete reboot of the system. 

Since the TMR system can be operated in the TMR mode or single processor 
mode, a test can be conducted to compare the efficiency of TMR hardware fault tolerance 
to software fault tolerance. One possible test could be to have the system count to a 
specified value while being subjected to the radiation beam. One test would use fault 


tolerant software in the single processor mode, while the other test would use software 


without fault tolerance in the TMR mode. The time that was required to complete the 
two operations could then be compared. 


af Conversion to Space Flight Board 


Once the system has been integrated and tested, the next logical step in its 
evolution would be to convert it into a space flight board. The new board could then be 
used to conduct further testing or as a computing node on the satellite. Some of the main 
topics that would need to be investigated for the conversion are listed below. 


e Determination of parts to be replaced with commercially available radhard 
models. 

Addition of an error detecting and correcting memory space. 

Addition of circuitry to detect SELs and reset the system. 

Weight and power analysis. 

Detailed timing analysis due to changes 1n system timing. 


4. Application to a State-of-the-Art (SOTA) Processor 


Although the R3081 microprocessor provides sufficient computational power to 
analyze fault tolerant techniques, its performance is far behind SOTA processors 
available today. One possible topic for future research on this project would be to 
basically start fresh with one of today’s SOTA processors. One of the limiting factors of 
TMR designs in the past has been the additional delay Jess in the communication paths 
by the logic needed to perform the voting. Because the processor cores have gotten so 
much faster than the memory devices that support them, the additional delay of the voters 


should be lost in the noise, but that will be left for someone else to prove. 
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APPENDIX A. TMR IMPLEMENTATION ORCAD SCHEMATICS 


This appendix contains the schematic for the TMR R3081 System. Figure A.1 is 
the top-level schematic and each block contained in it refers to one or more other figures. 


Table A.1 lists the figures in this appendix and the page they appear on. 


123 
Figure FGio UConGomllAaiSenormtic 


Table A.1. TMR R3081 System Schematics By Page Number 
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APPENDIX B. WINCUPL FILES 


When using programmable devices, such as Programmable Logic Devices (PLDs) 
and Field Programmable Gate Arrays (FPGAs), files must be written that instruct the 
device how to function when signals are applied to it. For the TMR R3081 design, a 
software program known as WinCUPL was used to write and compile the design files for 
the PLDs. WinCUPL also offers a simulation tool that allows the designer to test the 
programming files before the device is actually programmed in order to determine a 
specified set of test vectors produce the expected outputs. Sections A and B of this 
appendix contain the program file and the simulation file, respectively, for the Memory 
Control PLD that was presented in Chapter IV. Sections C and D contain the program 
and simulation file for the Memory Enable PLD that was also presented in Chapter IV. 


Ie MEMORY CONTROL PLD PROGRAM FILE 


Name MemCont ; 
PartNo ATF22V10C-7PC ; 
Date A730700>7 


Revision 01 ; 

Designer David Summers ; 
Company NPS ; 

Assembly TMR R3081 ; 
Location U54 ; 

Device Be2V0 


[RRR RRR RR RR RR RR RK CONSTANT DEFINITIONS tk RR Rk ko / 
SDEFINE LOW ‘B’0 
SDEFINE HIGH ‘B’l1 


/* kak keke kK KKK KKK KKK KKKRKKEKE INPUT PINS kaekkeKekkeke Keke KKK KK KKK KR EK * / 
Pin 1 = SYSCLK; (peo TOCEK. FR CRUA ys 
PIN 2 = INTCSN; 7 SUNT ERRUPT CHIP SELECT ae A 
PIN 3 = RAMCSN; [ oe CHL Sep nCr a7, 
Jee 4 = TIMERCSN; / * 2M ER CHiPs SELECT Ta 
PrN 5 = UARTCSN; P= mueRT CHIP SELECT xf 
PIN 6 = EPROMCSN; fae Oe OM CHIP SELECT tf 
PIN 7 = VOTBURSTN; /* VOTED BURST READ|WRITE NEAR i 
PIN 8 = VOTRDN; /* VOTED READ es 
PIN 9 = VOTWRN; /* VOTED WRITE a 
PINe 10> =-CVOUBER: /* CONTROL VOTER ERROR sid | 
PIN 11 = DVOTERR; /* DATA VOTER ERROR “i | 
PIN 13 = RESETN; /* SYNCHRONOUS SYSTEM RESET? ae 
PIN 19 = AVOTERR; /* ADDRESS VOTER ERROR os 
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Je KKK KKKKKKKKKKKKKKKKKKKEK 
fila 5 5 )cc6 7 | 
CYCENDN; 
BUSERRN; 


PIN 
PIN 
Pan 
PE 
Ean, 
PIN 


/* kKkekeKkkekkekeKekkkekekkKekKekeKeKke 


COUNTO. 


COUNT1 


COUNT2. 


COUNT3 


COUNTO 


COUNT! . 
COUNT 2 . 
COUNTS . 


COUNTO. 
GOUNEL . 
GOUNT2'. 
COUNTS . 


COUNT O:. 
COUNT. 
COUNT 2. 
COUNTS’. 


FIELD CNTR = 


18 


nou uw ue a 


VO@RInTN.D 


VOTINTN.AR 
VOTETINT SP 


ACKN; 
RDCENN ; 


VOTINTN; 


= !((AVOTERR # CVOTERR # DVOTERR # 


LOW; 
LOW; 


nt ou 


RESET OR CYCLE ENDSIGNAL. 
REFERENCE CLOCK. 


KKK KK KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KEKE KKEKKKKKKKKKKK 


HAPPENS IN ONE OF THE FPGAS. 
IS GENERATED BY THE ADDRESS DECODER. 


KKK KKK KKK KKK KKH EEK KKK KKK KKK KKK KKK KKK KKK KKK KEK KKH KEKE EKESESE 


[COUNTS . Gar: [a 
Tes 
/* 
/* 
/* 
/* 


LOGIC EQUATI 


OUTPUT PINS 


KEEKKKKKKKKKKKKKKK KK KKK 
WAIT STATE GENERATOR i 
CYCLE END SIGNAL bal 
BUS ERROR SIGNAL TO CPU a’ 
ACK SIGNAL TO CPU *7 
READ CLOCK ENABLE TO CPU sa: 
VOTER INTERRUPT TO €PuU ba 


kaekkekeKekkKeKkkekeKekekekkekkke kkk 


ONS 


KKK KK KKK KK KKK KEKEKEKKEKEKE KKK KEKE KEKKEKEKEKKEEKKKKKKKKKKKKKKKKKRKRKKKRKEK 


WAIT STATE COUNTER COUNT[3. .0] 


DHE PURPOSE OF THE WAIT STATE COUNTER I5 70 PROVIDE THe 
REFERENCE FOR THE MEMORY CONTROLLER SIGNALS. 
WHEN A READ OR WRITE CYCLE IS INITIATED AND RESETS WHEN THERE IS A 
THE COUNTER USES THE SYSCLK AS ITS 


IT STARTS COUNTING 


D = RESETN & CYCENDN & (!VOTRDN # !VOTWRN) & 
(COUNTO $ HIGH); 

.D = RESETN & CYCENDN & (!VOTRDN # !VOTWRN) & 
(COUNT): S COUNTS); 

D = RESETN & CYCENDN & (!VOTRDN # !VOTWRN) & 
(COUNT2 $ (COUNT1 & COUNTO)); 

.D = RESETN & CYCENDN & (!VOTRDN # !VOTWRN) & 
(COUNT3 $ (COUNT2 & COUNT1 & COUNTO)); 

.OE=HIGH; 

OE = HIGH; 

OE = HIGH; 

OE = HIGH; 

AR = LOW; 

AR = LOW; 

AR = LOW; 

AR = LOW; 

SP = LOW; 

SP = LOW; 

SP = LOW; 

SP = LOW; 

[COUNTS, COUNT2, CO@HTI,SCOUNTO] ; 


KM KKK KEKE KKK KKK KKK KKK KKK KKK KKK KKK KKK KKK KEK KEKE EKKEKKKKEKKKKK 


VOTER INTERRUPT SIGNAL 
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THE VOTER INTERRUPT SIGNAL INFORMS THE CPU WHEN A MISCOMPARE 
THE SIGNAL IS HELD UNTIL A INTCSN 


!'VOTINTN) & INTCSN) ; 


ay 


| il REKKKEKK KKK KKK KKK KK KEKE KE KKK KEK KK KEK KEK KEKE KEKE KEKE KRKKKK KEK KRKRKKKR KKK KK KKK KKKKE */ 


/> Sy 
fed CYCLE END SIGNAL x: / 
Tz */ 
/* THE PURPOSE OF THE CYCLE END SIGNAL IS TO SIGNAL THE END OF THE = 
/* OF THE CURRENT BUS CYCLE IN ORDER TO RESET THE COUNTER. THIS */ 


/* RESEiSellaslLe BY SeNGEUDENG A REFERENCE TO ITSELF IN THE EQUATIONS. */ 


/[* KR KK KKK KKK KEKE KEK KKK KKK KEK KEKE KKEKKE KK KKK KKRKRKRKRKRKR KK KKK KEK KKK KK KKK K KK KKK KK * / 


CYCENDN.D = !(RESETN & CYCENDN & ( 
('RAMCSN & (CNTR : ‘H’0OQ) & !VOTRDN & VOTBURSTN) 
# (!RAMCSN & (CNTR : ’H’6) & !VOTRDN & !VOTBURSTN) 
# (!RAMCSN & (CNTR ‘H’O) & !VOTWRN) 
# (!EPROMCSN & (CNTR 2 “HH 1) (eS VOTRDN & VOTBURSIN) 
# (!EPROMCSN & (CNTR : ’H’7) & !VOTRDN & !VOTBURSTN) 
# (!UARTCSN & (CNTR : ‘H’0O) & VOTBURSTN) 
* ('TOIMBRCON & (CNTR =: ‘'H*)}e& VOTBURSTDN) 


4 ()ENECSN ee (CNTR : ‘H’O) & VOTBURSTN) 
+ (CNIRG 2H’ F) 
) 


/* kaekkek keke keke KR KKK KKK KEK KK KKK KKK KKK KKK KKK KK KKK KKKKKKKR KK KKK KKKKKRKEKR KKK KEK = / 


/* ie 
/* READ CLOCK ENABLE */ 
/* sat 
/* THE READ CLOCK ENABLE SIGNAL IS USED BY THE CPU TO STROBE DATA */ 


/* OFF THE DATA BUS INTO ITS READ BUFFER. THIS SIGNAL IS STROBED ONE */ 
/* TIME FOR SINGLE READS AND FOUR TIMES FOR QUAD READS. ONLY RAM AND */ 
/* EPROM MEMORY USE THE QUAD WORD READS. aie f 


/* KEK KK KKK KEK KEK KKK KEKE KEK KKK KEK KKK KEKE KEKKKKKKEKEKKEKKKKKKKRKKRKKKREK */ 


RDCENN.D = !(RESETN & CYCENDN & !VOTRDN & ( 


(!RAMCSN & ( 
(ENTR == oO} 
# (!VOTBURSTN & (CNTR : ‘H’2)) 
m (?PVOTBURSTIN & (CNTR : ‘H*4)) 
nm (*VOTBURSTN & (CNTR : ‘H’6)) - 
) 


—— 


# (!EPROMCSN & ( 
(CNTR 3) Heer) 
me ()VOMeBURSEN & {CNTR =: “H'3)) 
" ('VOTEDRGIN & (CNTR : ’H’5)} 
7 ('VORBURSIN & (CNTR : °H°7)}) 
) 


) 

+ (1UARTCSHe(GnNmR : ‘H’0Q)) 
"+ ('TIMERGSNeeeveNTR : ‘’H’0)) 
7 (UINTCSN@ereniR : *’H’O)) 


7 


[ies ka kk Ka KKe Ke KR KKK K KK KKK KKKKKRKAKREKKKKKRKKKKKKKKKKKKRKKKKKKKKKKK KKK KKK KK KKK K 


/* ACKNOWLEDGE 


/* THE ACKNOWLEDGE SIGNAL IS USED BY THE MEMORY SYSTEM TO LET THE 

/* CPU KNOW THAT IT HAS PROCESSED THE WRITE CYCLE SUFFICIENTLY “AND 
/* THE GPU MAY°MOVE ON TO THE NEAT CYCLE. THIS S#GNAL IS GEERT ee 
/* IMPLICITLY ON SINGLE DATUM READS AND NO SOONER THAN FOUR CLOCKS 
/* BEFORE THE END OF THE LAST READ FOR BURSTS. 


/* Kaka Keke KKK KK KKK KKK KKK KKKKKKRKKKK KKK KK KK KKK KKKRKKRKKKRK KKK KKK K KKK KKKKKKKEK 


ACKN.D = !(RESETN & CYCENDN & ( 
(!RAMCSN & !VOTWRN & (CNTR : ‘’H’Q)) 


# (!RAMCSN & !VOTRDN & !VOTBURSTN & (CNTR : ’H’3)) 
# (!EPROMCSN & !VOTRDN & !VOTBURSTN & (CNTR : ‘’H’4)) 
# (!UARTCSN & !VOTWRN & VOTBURSTN & (CNTR : ’H’0O)) 
# (!TIMERCSN & !VOTWRN & (CNTR : ‘H’0O)) 
# (!INTCSN & !VOTWRN & (CNTR : ’H’0O)) 
) 
or 
ACKN.AR = LOW; 
ACKN.SP = LOW; 
/* Keka KKK KKKKKKEKKEKKKKKKKKKKKKRKKeKKKKKKKRKRKRKK KK KK KKK KKK KKK KK KKK KKK KKK KEK 
/* 
fr BUS ERROR 
/* 


/*"=THe BUS ERROR SIGNAL TS USED BY THE PROCESSCR TO END A BUS CYGEE 
/* THAT TRIES TO ACCESS AN ADDRESS THAT IS NOT POPULATED IN THE 
/* MEMORY SPACE. 


T= kaekkeKeKeaeKkeKaKkKeKeKeKKeK KKK KKK KKK KKK KKRKRKEKKK KK KKK KKKRKRKRKKKKKKKKKKKKKKKKKKKEKS 


BUSERRN.D = !(RESETN & CYCENDN & (CNTR : ‘H’F)); 
BUSERRN.AR = LOW; 

BUSERRN.SP = LOW; 

END; 
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z. 


MEMORY CONTROL PLD SIMULATION OUTPUT FILE 


CSIM(WM): CUPL Simulation Program 
Version 5.0a Serial# 10000000 


Gepyright 


(c) 1983, 1998 Logical Devices, Inc. 


CREATED Sat May 27 09:01:57 2000 


LISTING FOR SIMULATION FILE: MEMCONT.si 


V 

ARP ©) 

EUPT CD A CB V 
SIRMARBVVVVRV CCCCYU RO 
YNAEROUQOOOEO OOOO0CS DT 
STMRTMRTTTTST UUUUEEACI 
CCCCCCSRWEEEE NNNNNRCEN 
LSSSSSTIDRRRIR | PPT EDREANT 
KNNNNNNNNRRNR 3210NNNNN 
CIiliITiITiTO00G  LLLL eae 
CO1l1I111100010 LLLAEAian 
COLLIIISMOOL0 §6LLLEAAAAH 
ClOITI1I11000L0 LLLAEsGaH 
Cl10OT1T11110010 LLLEARAAE 
Cil0117100010 LEEAEAeAA 
CLLOLIITIIGOL0 6 CL LLLL EAA 
C1Ii10O1I1T00010 LUBREALEA 
CiLTO1III0010 CLCLAARad 
COLLIITZ010CIO” LULEEAALA 
COl1111110010  LLLLABAAE 
C101111010010 LLLHLHHLH 
C10TTiw110010™~ LELEHEEAE 


WOAH U PWD FP 


Name MEMCONT ; 
PartNo ATF22V10C-7PC; 
Date 5720/00; 


Revision 01; 

Designer DAVID SUMMERS; 
Company NPS; 

Assembly TMR R3081; 
Location U54; 

Device p22v10; 


[RRR TREK REEE RRR KER KN KE ER eee ee A / 


/* This device generates the memory control and interrupt ‘| 
/* signals required for memory reads and writes. It also ay 
/* generates the voter error interrupt signal. a 


J 0h Xe eee eR RRR RR RR KKK RRR ee a a 


/* Allowable Target Device Types: PLD22V10 / 


[RRR RARER EAR RRR ERR RE RRA R RR RRR EK RR ee ee ee ee / 


FIELD CNERe= [(COUNT3, COUNT2Z, COUNT, COUNTO]; 

ORDER: SYSCLK, INTCSN, RAMCSN, TIMERCSN, UARTCSN, EPROMCSN, 
VOTBURSTN, VOTRDN, VOTWRN, CVOTERR, DVOTERR, RESETN, AVOTERR, %2, 
COUNT3 ..COUNTO, CYCENDN, BUSERRN, ACKN, RDCENN, VOTINTN; 


— eet ee ee ee es ee ee ee ee ee ee es ws i oe oe 
ee ee ee ee ce ee ee ee ee ee ee ee ee eee es oe 


ee 


me ce ce eee ee ce ee ee ee es ee ee ee ee ee ee es es es es es es es ee ee i ei 
es ee ee ed 


Vv 

LES 
PUPT . CDLA 
SIRMARBVVVVRV 
YNAEROUOOOOEO 
STMRTMRTTITST 
CCCCECSRWEEEE 
LSSSSSTDRRRTR 


GrTOtALOPUOLO 
C1107 20010 
Gi710110100n9 
ClVTOLT Treo 
C€111101010010 
oral OG oko ne ale, 
C111101110010 
GTO11 10010610 
C101110010010 
€101110010010 
€101110010010 
ClO PET ooLoolo 
E1OLTTIOOLOO10 
C€101110010010 
Crorr100LOeLo 
CLlITTOOCLOORG 
C111100010010 
C111100010010 
C111100010010 
C1111 00040010 
Cll TAO CCLCOLG 
CLITTOCOLOOTO 
CiriiecerJoro 
C1111 0002G01Te 
Cir Pprreroor0 
GA ere ree O10 
C111111010010 
Cri rorooLo 
ei arorgoO10 
Si hiro Looro 
CeO O10 
CIT iar rO1ro010 
Cilia Oreo 
GliTiiroert0010 
Giri To10010 
Cli TiO ToOO10 
Cilia 1O1G010 
Cli OLCOTT O 
Cadi LOO 
CrlTiitroreo1o 
Citi ToOLco10 
Ga aes Ba Ral a0 (ONO Ne) 
(aa a Gs bps Gs Fos En 8. 
Coliiii1o0010 
SOT Tir 0010 
Sis PaO. 
eOlITEVEOLOOLO 
SO Pia rOoL0 
Gia tT 110011 
Guar it lt Loo10 


CB WV 
CECeYUSRO 
O000CS DT 
UUUUEEACT 
NNNNNRCEN 
TTTIDRENT 
3210NNNNN 


LLLHLHHLH 
LLLLHHHHH 
LLLHLHHLH 
LLLLHHHHH 
LLLHHHHHH 
LLHLLHHLH 
LLLLHHHHH 
LLLHHHHLH 
LLHLHHHHH 
LLHHHHHLH 
LHLLHHLHH 
LHLHHHHLH 
LHHLHHHHH 
LHHHLHHLH 
LLLLHHHHH 
LLLHHHHHH 
LLHLHHHLH 
LLHHHHHHH 
LHLLHHHLH 
LHLHHHLHH 
LHHLHHHLH 
LHHHHHHHE. 
HLLLLHHLH 
LLLLHHHHH 
LLLHHHHHH 
LLHLHHHHH 
LLHHHHHHH 
LHLLHHHHH 
LHLHHHHHH 
LHHLHHHHH 
LHHHHHHHH 
HLLLHHHHH 
HLLHHHHHH 
HLHLHHHHH 
HLHHHHHHH 
HHLLHHHHH 
HHLHHHHHH 
HHHLHHHHH 
HHHHHHHHH 
LLLLLLHHH 
LLLLHHHHH 
LLLLHHHHH 
LLLLHHHHL 
LLLHLHLHH 
LLLLHHHHH 
LLLLHHHHL 
LLLHLHHLH 
LLLLHHHHH 
LLLLHHHHL 
LLLLLHHHH 
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a. MEMORY ENABLE PLD PROGRAM FILE 


Name MemEn ; 

PartNo ATF22V10C-7PC; 
Date ay 30/00; 
Revision 01; 

Designer David Summers; 
Company NPS; 

Assembly TMR R3081; 
Location U55; 

Device 2 2570 ; 


/* kKaekKkekekekeKeKeKK KKK KKK KK KKKK 


INPUT PINS 


Kak KKK KKK KKK KK KKK KK KKK 


Pal 1 = SYSGEK; /* SYSCLK FR CPUA 

PIN 2 = VOTRDN; /* VOTED READ 

PIN 3 = VOTWRN; /* NOTED WRETE 

PIN 4 = CYCENDN; /* CYCLE END SIGNAL 
PIN 6 = VOTBEON; /* VOTED BYTE ENABLE 0 
PIN 7 = VOTBEIN; /* VOTED BYTE ENABLE 1 
PIN 8 = VOTBEZN, /* VOTED BYTE ENABLE 2 
PIN 9 = VOTBE3N; /* VOTED BYTE ENABLE 3 
PIN 13 = SYS_RESETN; j/* ~SYS@PEMTRESEA: 


I ai kKaekekkKeKe Ke KKK KK KK KKK KKK KK 


PIN 14 = RDDATAENN; /* READ DATA ENABLE FOR XCEIVER ENABLE 
PIN 15 = WRDATAEN; /* WRITE DATA ENABLE FOR XCEIVER ENABLE 
PIN 16 = RDENN; /* READ ENABLE 

PIN 17 = WRENDN; /* WRITE ENABLE FOR BYTE 3 

PIN 18 = WRENCN; /* WRITE ENABLE FOR BYTE 2 

PIN 19 = WRENBN; /* WRITE ENABLE FOR BYTE 1 

PIN 20 = WRENAN; /* WRITE ENABLE FOR BYTE 0 

PIN 22 = RESETPOS; /* SYNCHRONOUS SYSTEM RESET FOR UART 
PIN 23 = RESETN; /* SYNCHRONOUS SYSTEM RESET 


f= kKkekeKkekeKeKe Keke KKK KK KKK 


30 


OUTPUT PINS 


CONSTANT DEFINITIONS 


kKaekkekkekekekekeKekeKeKeKe KKK KKK KK 


kakkkekeekkkeke kee RRR KEK 


SDEFINE LOW 


SDEPINE HiGH) Bi 


f= kaerkkekekekeKe keke Ke KKK KKK KKK KK kakkkeekekekekekeke kK KKK KKK KK 


LOGIC EQUATIONS 


Pes kKaekkkekkkeKekekekeKe KKK KKK KKK KK KKK KKK KKK KK KKK KKK KK KKK KR KKK KK KK KKK KK KKK 


/* THE READ ENABLE STROBE IS USED TO ACTIVATE THE OUTPUT ENABLES ON 
SINCE ALL READS ARE DONE 32 BITS WIDE 
/* THERE IS NO NEED TO DIFFERENTIATE BETWEEN THE BYTES ON THE BUS. 

/* THE SIGNAL IS INITIATED BY THE VOTED READ AND DISABLED BY THE 


/* THE MEMORY AND 1/70 DEVECES. 


/* CYCLE END SIGNAL. 


/* KKEKKKKKEKKEKEKEKKEKEKKKEKKEKE KKK KKK KKK KKK KK KKK KKK KR KKK KKK KKK KKK KKK KKK KK KKK K 


RDENN.D =! (SYS_RESETN & 


RDENN.AR 
RDENN.SP 


LOW; 
LOW; 


READ ENABLE STROBE 


Y'VOURDN & CY@GENDN); 


—d 
a 

» ~~ 

——" 


i KKK KKKRKKKKKEKRRKREKREKKKKEKRKRKKKR KKK KKK KKK KKKKKRKK KKK KKKRKK KKK KKK KKK K 


j= WRITE ENABLE STROBES 


/* THE WRITE ENABLE STROBES DETERMINE WHICH BYTES OF THEes2 Bia DaTA 
/* BUS WILL BE INVOLVED IN THE WRITE CYCLE, SINCE THE R30USitaeai 

/* PERFORM PARTIAL WORD WRITES. THE WRITE ENABLE STROBES CORRESPOND 
/* TO THE BYTES ON THE DATA BUS AS FOLLOWS: 


1 WRENAN -> DATA(7..0) 

po WRENBN -> DATA(1i5..8) 
‘ail WRENCN -> DATA(23..16) 
fs WRENDN -> DATA(31..24) 


/* THE !VOTWRN AND !VOTBE_N SIGNALS INITIATES THE STROBE AND THE 
/* CYCENDN SIGNAL DEACTIVATES THEM. ONE TO ALL OF THE STROBES CAN BE 
/ Shelve DURING THE WRITE CYenE.. 


7 Sas ka Km KKK KKK KEKKKKEEKEKEKKKEKKKEKKKKKEKKEKEKEKKKKEKEKKKKEKKEKKKEK KKK KKK KKKKKKRKRKK 


WRENAN.D = !(SYS_RESETN & (!VOTWRN & !VOTBEON & CYCENDN) ); 
WRENBN.D = !(SYS_RESETN & (!VOTWRN & !VOTBEIN & CYCENDN) ); 
WRENCN.D = !(SYS_RESETN & (!VOTWRN & !VOTBE2N & CYCENDN) ); 
WRENDN.D = !(SYS_RESETN & (!VOTWRN & !VOTBE3N & CYCENDN) ) ; 
WRENAN.AR = LOW; 
WRENBN.AR = LOW; 
WRENCN.AR = LOW; 
WRENDN.AR = LOW; 
WRENAN.SP = LOW; 
WRENBN.SP = LOW; 
WRENCN.SP = LOW; 
WRENDN.SP = LOW; 


Vi 3 KaeKKKRKRKKKEKRKKKEKKKEKKEKREKKEKEKKREKKEEKKEKKKKEKEKEKKK KKK KKK KK KKKKR KKK KKK KKKRKEK 


es READ DATA ENABLE AND WRITE DATA ENABLE STROBES 


/* THE READ DATA ENABLE AND WRITE DATA ENABLE STROBES DETERMINE THE 

/* DIRECTION DATA TRAVELS THROUGH THE DATA VOTER FPGA AND THE 

/* VJ4AHCT623 BUS TRANSCEIVER. THE READ AND WRITE SIGNALS COME ON TO 
{/"SBARLY IN THE’ CYCLE AND STAY ON TOO LATE IN THE CYCEE’ TO BE Uner:. 

/* THESE SIGNALS PREVENT BUS CONTENTION WHEN DEVICES WITH LONG TURN 

/* OFF TIMES DRIVE THE BUS JUST PRIOR TO IT BEING TURNED AROUND GND 

/* DRIVEN BY THE PROCESSOR: 


/* KaEKKKRKKRKEKREKKKEEKEKEKEKEKKKREKKKEKEKEKEKRKEKKE KKK KEKKEKEKK KE KKK KKK KR KKK KKK KKKEKK 


WRDATAEN.D = SYS_RESETN & !VOTWRN & CYCENDN; 
RDDATAENN.D = ! (SYS_RESETN & !VOTRDN & CYCENDN) ; 


WRDATAEN.AR = LOW; 
RDDATAENN.AR = LOW; 


WRDATAEN.SP = LOW; 
RDDATAENN.SP = LOW; 


/* KEK KEK KKK KKK KEKE KEK KKK KKK KEKE KKK KKK KEKE KKK KEKE KEKE KEK KKK KEKE KEKE KEKE KKK KEKE KE KEKE 


Ts RESETPOS AND RESETN 


/* THESE TWO SIGNALS PROVIDE SYNCHRONOUS POSITIVE AND NEGATIVE LOGIC 
/* RESETS FOR THE SYSTEX. 


/* KKK KKKEK KKK KK KKK KKK KKK KKK KKK KKEKEKEKEKEKK KKK KEKE KKK EK KKK KEKE KK KKK KK KKK KKK 


RESETPOS.D = !SYS_RESETN; 
RESETN.D = SYS_RESETN; 


RESETPOS.AR = LOW; 
RESETN.AR = LOW; 


RESETPOS.SP = LOW; 
RESETN.SP = LOW; 


END; 
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4. MEMORY ENABLE PLD SIMULATION FILE 


CSIM(WM): CUPL Simulation Program 

Version 5.0a Serial# 

Copyright (c) 1983, 1998 Logical Devices, Inc. 
CREATED Fri May 26 13:19:41 2000 


LISTING FOR SIMULATION FILE: MEMEN.s1 


Name MEMEN ; 

PartNo ATF22V10C-7PC; 
Date 5/267.007 
Revision 01; 

Designer DAVID SUMMERS; 
Company NPS; 

Assembly TMR R3081; 
Location U55; 

Device D22y 10; 


WOMANI HA WM PWN 


dl heal bee PRR RRR REREAEREARAR AE RKEE REAR EERE RRR KE RR RAK RRR EE ERE RAS Xe ee 


12: /* This device generates the memory write byte enable signals bad 
13: /* and positive and negative logic synchronous reset signals. 7, 
14: /* It also generates the read and write data enable strobes that*/ 
15: /* control the tri-state output buffers on the bus transceivers */ 


cee, and EPGAs. * / 
‘Wjae ptr RR EAA RE K KARR RAK RARE RRR RR ERE RAK Ss EE ee 
18: /* Allowable Target Device Tyres: PLD22V10 7 
19: [RR EREEREEKERERREAERKEEREEKRE KKK KEKE KKKEKEKE KER RRREERE SE We 
#4 Oe 


21: ORDER: SYSCLK, SYS_RESETN, VOTRDN, VOTWRN, CYCENDN, %1, VOTBEON, 
VOTBE1N, VOTBE2N, VOTBE3N, %1, WRDATAEN, RDDATAENN, %1, RDENN, $1, 
WRENDN, WRENCN, WRENBN, WRENAN, %1, RESETPOS, RESETN; 


— = a= aes ame ame cee cee wee Cm cee cme cow Oe ee ee ce oe ew cow mm me we we oe we ee om ow ee = oa 
— == ae ame ame ame eee cee cer cme com cee fee ee ee cee oe oe ame ew om cme cee se om ome ee =e ow ew oe 


YEOOC THT AT R RRRR EE 
SSTTE BBBB TA D BBEE TS 
CERWN EEEE AE E NNNN PE 
LTDRD 0123 EN N DCBA OT 
KNNNN NNNN NN N NNNN SN 
Cele "GOll1 11i11i°Ge HieneAaH HL 
Oz GLoOl Titi Hb, Lb Hatt LH 
OoGs; Cl000 1111 LH H HHH LH 
C004: C1110 1111 LH H HBR LH 
Cole. Gelil 1111 LH A HHH LH 
GoGs: Cil0Ol 0000 HH H LLLL LH 
OOC7-e2.00 0000 LH H HHHH LH 
0008: C1101 0000 HH H LLLL LH 
OOCeeeer Ol Lill ne Wo Hea LH 
COlCG= GO101 1111 DoH REA AL 
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APPENDIX C. XILINX FOUNDATION DESIGNS 


Just as with the PLDs, the FPGAs have to be programmed. Although the 
WinCUPL software had provisions for programming Xilinx FPGAs, it required the files 
to be written in HDL or Verilog. An alternative offered by the Xilinx Foundation 
software is schematic programming of the FPGA. The designer enters the design in a 
schematic format and the software converts the file to a netlist, which is compiled into the 
appropriate format for programming the FPGA. Section A of this appendix contains the 
Foundation Schematics for the design of the Address Voter FPGA and Section B contains 
the schematics for the Data and Control Voter FPGA. Each of these FPGA designs were 
presented in Chapter IV. 


1. ADDRESS VOTER FPGA 


The Address Voter FPGA is responsible for performing a majority vote on the 
three address busses, decoding the upper thirteen bits of the address bus into chip select 
signals, and providing the system timer interrupt. The design of this FPGA is presented 


in the following figures. Table C.1 lists the figures presented in this section. 


Figure Number and Description 
re C.1. Address Voter FPGA Top Level Schematic 
re C.2. 18-Bit Counter_ 


re C. 5. CPU B Address Bus a Specification 
Ee re C.6. CPU C Address Bus Input Specification 
Figure C.7. Voted Address Bus Output Specification 


Table C.1. Address Voter Figures By Page Number 
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LOG=P2 
LOC=P5 
LOC=P8 
LOC=P11 
LOC=P15 
LOC=P18 
LOC=P23 
LOC=P26 
LOC=P31 
LOC=P34 
LOC=P38 
LOC=P42 
LOC=P46 
LOC=P49 
LOC=P52 
LOG=P55 
LOC=P63 
LOC=P66 
LOC=P69 
LOG=F72 
LOC=P76 
LOC=P79 
LOC=P84 
LOC=P87 
LOC=P93 
LOC=P96 
LOC=P100 
LOC=P104 
LOC=P108 
LOC=P111 









INPUT2 
INPUT3 
INPUT4 
INPUTS 
INPUT6 
INPUT? 
INPUTS 
INPUTS 
INPUT10 
INPUT11 
INPUT12 
INPUT13 
INPUT14 
INPUT15 
INPUT16 
INPUT17 
ein 8 
INPUT19 


INPUT20 








INPUT21 
INPUTZ2 
INPUT23 
INPUT24 
INPUT25 
INPUT26 
INPUT27 
INPUT28 
INPUT29 
INPUT30 


INPUT31 


INPUT[31:2] 





David Summers 


'|Naval Postgraduate School 


TMR R3081 Project 
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Project: ADDRVTR 


Macro: INBUSA 


Date: 05/27/100 


LOC=P3 
LOC=P6 
LOC=PS 
LOC=P12 
LOC=P16 
LOC=P20 
LOC=P24 
EOC@=—27 
LOC=P32 
LOC=P35 
EOG=R339 
LOC=P43 
LOC=P47 
LOC=P50 
LOC=P53 
LOC=P56 
LOC=P64 
LOC=P67 
LOC=P70 
LOC=P73 
LOC=P77 
LOC=P81 
LOC=P85 
LOC=P88 
LOC=P94 
LOG=P97 
LOC=P102 
LOC=P105 
LOC=P109 
LOC=P112 


INPUT2 

INPUT3 

INPUT4 

INPUTS 

INPUT6 

INPUT7 

INPUTS 

INPUTS 

INPUT10 
INPUT11 
INPUTi2 
INPUT13 
INPUT14 
INPUT15 
INPUT16 
INPUT17 
INPUT18 
INPUT19 
INPUT20 
INPUT21 
INPUT22 
INPUT23 
INPUT24 
INPUT25 
INPUT26 
INPUT27 
INPUT28 
INPUT29 
INPUT3O 


INPUT31 


INPUT[31:2} 


David Summers Project: ADDRVTR 
Naval Postgraduate School Macro: INBUSB | 
TMR R3081 Project Date: 05/27/100 
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LOC=P4 
LOC=P7 
LOC=F10 
LOG=P13 
LOG=P 17 
LOC=P21 
LOC=P25 
LOC=P28 
LOC=P33 
LOC=P36 
LOC=P41 
LOC=P44 
LOC=P48 
LOC=P51 
LOC=P54 
LOC=P57 
EOC=F65 
LOC=P68 
EOC=E71 
LOC=P74 
LOC=P78 
LOC=P82 
LOC=P86 
LOC=P92 
LOC=P95 
LOC=P99 
LOC=P103 
LOC=P107 
LOC=F110 
EOC=F ania 


SHORRRE HCO COO UCE PEER NKKKEE 


ie INPUT2 a 
IBUF 

Pes INPUT3 4 
IBUF 

[Ss INPUT4 , 
IBUF 

bE INPUTS 4 
IBUF 

cS INPUT6 e 
IBUF 

P= INPUT7 2 
IBUF 

Ps INPUTS 4 
IBUF 

es INPUTS A 
IBUF 

[= INPUT10 4 
IBUF 

ee INPUT11 4 
IBUF 

> INPUT12 e 
IBUF 

D> INPUT13 4 
IBUF 

> INPUT14 4 
IBUF 

SS INPUT15 y 
IBUF 

Pe INPUT16 4 
IBUF 

> INPUT17 A 
IBUF --_ 

eS INPUT18 y 
IBUF 

ES INPUT19 y 
iIBUF 

> INPUT20 4 
IBUF . 

[s ; INPUT21 a 
IBUF ; 

ies INPUT22 y 
IBUF 

Py INPUT23 4 
IBUF 

[= INPUT24 4 
IBUF 

Ds INPUT25 4 
IBUF 

> INPUT26 4 
IBUF 

[Ss INPUT27 4 
IBUF 

BS INPUT28 4 
IBUF 

SS INPUT239 4 
iIBUF 

ee INPUT3O - 
IBUF 

INPUT31 4 

IBUF 


INPUT[31:2] 






David Summers 






Naval Postgraduate School 
TMR R3081 Project 


halt 





Project: ADDRVTR 


Macro: INBUSC 
Date: 05/27/100 


OUTPUT[17:2] 











> OPAD LOC=P192 
. OPAD LOC=P193 
. LOC=P194 
> CUTPUTS LOC=P197 
> —— OUTPUTS LOC=P198 
oe nalins LOC=P199 
> -OUTRUTS LOC=P200 
p> OUTEUTS LOC=P202 
> OUTPUTIO ae LOC=P203 
> En = LOC=P205 
. LOC=P206 
. LOC=P207 
. LOC=P208 
. LOC=P209 
> OPAD LOC=P210 
> OUTPUTI7 : LOC=P213 


David Summers Project: ADDRVTR 


Naval Postgraduate School Macro: OUTVOT 


TMR R3081 Project Date: 05/27/100 





2, DATA AND CONTROL VOTER FPGA 


The Data and Control Voter FPGA performs a three bit majority vote on the Data 
and Control Busses of the three microprocessors. The Data Bus is a 32-bit wide bus and 
the Control Bus is an 8-bit wide bus. Because the Data Bus is bidirectional, the FPGA 
must be able to vote processor data to memory on writes and pass memory data back to 
all three processors on reads. This FPGA incorporates both 4-bit wide voters and 8-bit 
wide voters in its design. The design of this FPGA is presented in the following figures. 


Table C.2 lists the figures presented in this section. 












Table C.2. Data and Control Voter Figures By Page Number 


Shs: 


Cwa_] 

CE) lOcer 

CE} LOC 
Dendd (usnerary qject OCTRLVTA 
nad Postyedete Schod i Meow DATACTALYTR 
er FS) « 06/7100 





Aeon g335 
“gag gags 
Jay BEE 


PEAS AP AS 8 ASAD AL AS 
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af 


a. 
| 
aes 

i 













| nt | ae 
a [4 
hz: = 
AO: 4 
peseelll 


, nt 
( 
‘ 
ta "a 
Saas 


Fos 








LOC=P127 
LOC=P130 
LOC=P133 
L@@=F137 


LOC=P126 
LOC=P129 
LOC=P132 
LOC=P136 


LOC=P125 
LOC=P128 
LOC=P131 
LOC=P134 
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TMR R3081 Project 
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INPUTC[3:0] 


INPUTB[3:0] 


INPUTA[3:0] 


Project: DCTRLVTR 
Macro: CTRL_IN 
Date: 05/28/100 


LOC=P147 
LOC=P152 
LOC=P155 
LOC=P159 


LOC=P146 
LOC=P149 
LOC=P154 
LOC=P157 


LOC=P145 
LOC=P148 
LOC=P153 
LOC=P156 
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INPUTC[3:0] 


INPUTB[3:0] 


INPUTA[3:0] 


Project: DCTRLVTR 
Macro: BEN_IN 
Date: 05/28/100 


DATAOUT]31:0} 


DATAIN(31:0] 


oaTaane 








LOCePS 
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Caleane 





. n i 

js ' 
E F ry 
. < 

















LOCeP63 


LOCe P66 


LOCeP69 


LOCeP72 


LOCeP76 


LOCeP79 


LOCeP84 


LOCsP87 


LOC=P93 


LOC=P96 


LOCeP100 


LOCeP104 


LOCeP108 


LOCeP111 


LOCeP114 


LOCa£P117 


DATAOUT[31:0} ours: © 
? 
DATAIN{31:0] = vou: =, 


















Da Taree oS 
CHE L000 ais SD toere 
5 —* <j 
as 
tr 
ml 
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Ca Taser 6 xt <1 
' 
mare 
“oe OC=P73 
CHE vocerz eee eT ce 
a 
carp 
CEE Locerrs ne ee! be) oe 
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Er 
ee ED toe 
CFS LOCeP20 bus ™ 
CIS > LOCaP24 ~— la J ee —_— 
aE 
t 
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| 
CoS LOCeP32 - 
= 
SED Loceas i OO 
CoS) LOceP39 a arias 
CHD Locepas __ LocaPi0s 
TSS LOCaPA7 dt el 
c= 
CBS LOCePsO , eae 
CRE LOCaPSS al niiitetea 
Cs > LOCaPS6 > LOCePit8 
Preect OCTRLVTR 
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DATAOUT]31:0} ou: D 
? 
Tt 






DATAIN(31:0] 

CN > LOCePs LOC=P65 

(e LOCaP7 LOCeP68 

C5 > LOC=P10 LOCaP71 
patane Ce > LOCePI3 LOC=P74 
= =e LocaPiz LOCaP78 
we _ Ce LOCsP21 LOC=P82 
= ‘~ CD LOCeP25 LOCePa6 
a = x Cr > LoceP28 LOC=P92 
on Zz Ce > Loc|P33 LOCeP9s 
= | nL OceESe LOCaPe9 
w | " Ce > LOCePa1 LOCeP103 
_ Co > LOCeP as LOCaP107 

Cee > LOCsP48 LOC=P110 
oo C D> LOCePS1 LOCaP113 
~ Co) LOCaPS4 LOCaP116 
a Ce > LOCaPS7 LOCaP123 
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VOTDATAOUT]3 1:0} 
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